Improving User Experience In Information Retrieval Using Semantic Web And Other Technologies

The need to find, access and extract information has been the motivation for many different fields of research in the past few years. The fields such as Machine Learning, Question Answering Systems, Semantic Web, etc. each tries to cover parts of the mentioned problem. Each of these fields have introduced many different tools and approaches which in many cases are multi-disciplinary, covering more than one of these fields to provide solution for one or more of them. On the other hand, the expansion of the Web with Web 2.0, gave researchers many new tools to extend approaches to help users extract and find information faster and easier. Currently, the size of e-commerce and online shopping, the extended use of search engines for different purposes and the amount of collaboration for creating content on the Web provides us with different possibilities and challenges which we address some of them here

Semantic multimedia modelling & interpretation for search & retrieval

With the axiomatic revolutionary in the multimedia equip devices, culminated in the proverbial proliferation of the image and video data. Owing to this omnipresence and progression, these data become the part of our daily life. This devastating data production rate accompanies with a predicament of surpassing our potentials for acquiring this data. Perhaps one of the utmost prevailing problems of this digital era is an information plethora. Until now, progressions in image and video retrieval research reached restrained success owed to its interpretation of an image and video in terms of primitive features. Humans generally access multimedia assets in terms of semantic concepts. The retrieval of digital images and videos is impeded by the semantic gap. The semantic gap is the discrepancy between a user’s high-level interpretation of an image and the information that can be extracted from an image’s physical properties. Content- based image and video retrieval systems are explicitly assailable to the semantic gap due to their dependence on low-level visual features for describing image and content. The semantic gap can be narrowed by including high-level features. High-level descriptions of images and videos are more proficient of apprehending the semantic meaning of image and video content. It is generally understood that the problem of image and video retrieval is still far from being solved. This thesis proposes an approach for intelligent multimedia semantic extraction for search and retrieval. This thesis intends to bridge the gap between the visual features and semantics. This thesis proposes a Semantic query Interpreter for the images and the videos. The proposed Semantic Query Interpreter will select the pertinent terms from the user query and analyse it lexically and semantically. The proposed SQI reduces the semantic as well as the vocabulary gap between the users and the machine. This thesis also explored a novel ranking strategy for image search and retrieval. SemRank is the novel system that will incorporate the Semantic Intensity (SI) in exploring the semantic relevancy between the user query and the available data. The novel Semantic Intensity captures the concept dominancy factor of an image. As we are aware of the fact that the image is the combination of various concepts and among the list of concepts some of them are more dominant then the other. The SemRank will rank the retrieved images on the basis of Semantic Intensity. The investigations are made on the LabelMe image and LabelMe video dataset. Experiments show that the proposed approach is successful in bridging the semantic gap. The experiments reveal that our proposed system outperforms the traditional image retrieval systems

Semantic multimedia modelling & interpretation for annotation

The emergence of multimedia enabled devices, particularly the incorporation of cameras in mobile phones, and the accelerated revolutions in the low cost storage devices, boosts the multimedia data production rate drastically. Witnessing such an iniquitousness of digital images and videos, the research community has been projecting the issue of its significant utilization and management. Stored in monumental multimedia corpora, digital data need to be retrieved and organized in an intelligent way, leaning on the rich semantics involved. The utilization of these image and video collections demands proficient image and video annotation and retrieval techniques. Recently, the multimedia research community is progressively veering its emphasis to the personalization of these media. The main impediment in the image and video analysis is the semantic gap, which is the discrepancy among a user’s high-level interpretation of an image and the video and the low level computational interpretation of it. Content-based image and video annotation systems are remarkably susceptible to the semantic gap due to their reliance on low-level visual features for delineating semantically rich image and video contents. However, the fact is that the visual similarity is not semantic similarity, so there is a demand to break through this dilemma through an alternative way. The semantic gap can be narrowed by counting high-level and user-generated information in the annotation. High-level descriptions of images and or videos are more proficient of capturing the semantic meaning of multimedia content, but it is not always applicable to collect this information. It is commonly agreed that the problem of high level semantic annotation of multimedia is still far from being answered. This dissertation puts forward approaches for intelligent multimedia semantic extraction for high level annotation. This dissertation intends to bridge the gap between the visual features and semantics. It proposes a framework for annotation enhancement and refinement for the object/concept annotated images and videos datasets. The entire theme is to first purify the datasets from noisy keyword and then expand the concepts lexically and commonsensical to fill the vocabulary and lexical gap to achieve high level semantics for the corpus. This dissertation also explored a novel approach for high level semantic (HLS) propagation through the images corpora. The HLS propagation takes the advantages of the semantic intensity (SI), which is the concept dominancy factor in the image and annotation based semantic similarity of the images. As we are aware of the fact that the image is the combination of various concepts and among the list of concepts some of them are more dominant then the other, while semantic similarity of the images are based on the SI and concept semantic similarity among the pair of images. Moreover, the HLS exploits the clustering techniques to group similar images, where a single effort of the human experts to assign high level semantic to a randomly selected image and propagate to other images through clustering. The investigation has been made on the LabelMe image and LabelMe video dataset. Experiments exhibit that the proposed approaches perform a noticeable improvement towards bridging the semantic gap and reveal that our proposed system outperforms the traditional systems

Does the way museum staff define inspiration help them work with information from visitors' Social Media?

Since the early 2000s, Social Media has become part of the everyday activity of billions of people. Museums and galleries are part of this major cultural change - the largest museums attract millions of Social Media 'friends' and 'followers', and museums now use Social Media channels for marketing and audience engagement activities. Social Media has also become a more heavily-used source of data with which to investigate human behaviour. Therefore, this research investigated the potential uses of Social Media information to aid activities such as exhibition planning and development, or fundraising, in museums. Potential opportunities provided by the new Social Media platforms include the ability to capture data at high volume and then analyse them computationally. For instance, the links between entities on a Social Media platform can be analysed. Who follows who? Who created the content related to a specific event, and when? How did communication flow between people and organisations? The computerised analysis techniques used to answer such questions can generate statistics for measuring concepts such as the 'reach' of a message across a network (often equated simply with the potential size of the a message's audience) or the degree of 'engagement' with content (often a simple count of the number of responses, or the number of instances of communication between correspondents). Other computational analysis opportunities related to Social Media rely upon various Natural Language Processing (NLP) techniques; for example indexing content and counting term frequency, or using lexicons or online knowledge bases to relate content to concepts. Museums, galleries and other cultural organisations have known for some time, however, that simple quantifications of their audiences (the number of tickets sold for an exhibition, for example), while certainly providing indications of an event's success, do not tell the whole story. While it is important to know that thousands of people have visited an exhibition, it is also part of a museum's remit to inspire the audience, too. A budding world-class artist or ground-breaking engineer could have been one of the thousands in attendance, and the exhibition in question could have been key to the development of their artistic or technical ideas. It is potentially helpful to museums and galleries to know when they have inspired members of their audience, and to be able to tell convincing stories about instances of inspiration, if their full value to society is to be judged. This research, undertaken in participation with two museums, investigated the feasibility of using new data sources from Social Media to capture potential expressions of inspiration made by visitors. With a background in IT systems development, the researcher developed three prototype systems during three cycles of Action Research, and used them to collect and analyse data from the Twitter Social Media platform. This work had two outcomes: firstly, prototyping enabled investigation of the technical constraints of extracting data from a Social Media platform (Twitter), and the computing processes used to analyse that data. Secondly, and more importantly, the prototypes were used to assess potential changes to the work of museum staff information about events visited and experienced by visitors was synthesised, then investigated, discussed and evaluated with the collaborative partners, in order to assess the meaning and value of such information for them. Could the museums use the information in their event and exhibition planning? How might it fit in with event evaluation? Was it clear to the museum what the information meant? What were the risks of misinterpretation? The research made several contributions. Firstly, the research developed a definition of inspiration that resonated with museum staff. While this definition was similar to the definition of 'engagement' from the marketing literature, one difference was an emphasis upon creativity. The second set of contributions related to a deeper understanding of Social Media from museums' perspective, and included findings about how Social Media information could be used to segment current and potential audiences by 'special interest', and find potential expressions of creativity and innovation in the audience's responses to museum activities. These findings also considered some of the pitfalls of working with data from Social Media, in particular the tendency of museum staff to use the information to confirm positive biases, and the often hidden biases caused by the mediating effects of the platforms from which the data came. The final major contribution was a holistic analysis of the ways in which Social Media information could be integrated into the work of a museum, by helping to plan and evaluate audience development and engagement. This aspect of the research also highlighted some of the dangers of an over-dependency upon individual Social Media platforms which was previously absent from the museums literature

Text-image synergy for multimodal retrieval and annotation

Text and images are the two most common data modalities found on the Internet. Understanding the synergy between text and images, that is, seamlessly analyzing information from these modalities may be trivial for humans, but is challenging for software systems. In this dissertation we study problems where deciphering text-image synergy is crucial for finding solutions. We propose methods and ideas that establish semantic connections between text and images in multimodal contents, and empirically show their effectiveness in four interconnected problems: Image Retrieval, Image Tag Refinement, Image-Text Alignment, and Image Captioning. Our promising results and observations open up interesting scopes for future research involving text-image data understanding.Text and images are the two most common data modalities found on the Internet. Understanding the synergy between text and images, that is, seamlessly analyzing information from these modalities may be trivial for humans, but is challenging for software systems. In this dissertation we study problems where deciphering text-image synergy is crucial for finding solutions. We propose methods and ideas that establish semantic connections between text and images in multimodal contents, and empirically show their effectiveness in four interconnected problems: Image Retrieval, Image Tag Refinement, Image-Text Alignment, and Image Captioning. Our promising results and observations open up interesting scopes for future research involving text-image data understanding.Text und Bild sind die beiden häufigsten Arten von Inhalten im Internet. Während es für Menschen einfach ist, gerade aus dem Zusammenspiel von Text- und Bildinhalten Informationen zu erfassen, stellt diese kombinierte Darstellung von Inhalten Softwaresysteme vor große Herausforderungen. In dieser Dissertation werden Probleme studiert, für deren Lösung das Verständnis des Zusammenspiels von Text- und Bildinhalten wesentlich ist. Es werden Methoden und Vorschläge präsentiert und empirisch bewertet, die semantische Verbindungen zwischen Text und Bild in multimodalen Daten herstellen. Wir stellen in dieser Dissertation vier miteinander verbundene Text- und Bildprobleme vor: • Bildersuche. Ob Bilder anhand von textbasierten Suchanfragen gefunden werden, hängt stark davon ab, ob der Text in der Nähe des Bildes mit dem der Anfrage übereinstimmt. Bilder ohne textuellen Kontext, oder sogar mit thematisch passendem Kontext, aber ohne direkte Übereinstimmungen der vorhandenen Schlagworte zur Suchanfrage, können häufig nicht gefunden werden. Zur Abhilfe schlagen wir vor, drei Arten von Informationen in Kombination zu nutzen: visuelle Informationen (in Form von automatisch generierten Bildbeschreibungen), textuelle Informationen (Stichworte aus vorangegangenen Suchanfragen), und Alltagswissen. • Verbesserte Bildbeschreibungen. Bei der Objekterkennung durch Computer Vision kommt es des Öfteren zu Fehldetektionen und Inkohärenzen. Die korrekte Identifikation von Bildinhalten ist jedoch eine wichtige Voraussetzung für die Suche nach Bildern mittels textueller Suchanfragen. Um die Fehleranfälligkeit bei der Objekterkennung zu minimieren, schlagen wir vor Alltagswissen einzubeziehen. Durch zusätzliche Bild-Annotationen, welche sich durch den gesunden Menschenverstand als thematisch passend erweisen, können viele fehlerhafte und zusammenhanglose Erkennungen vermieden werden. • Bild-Text Platzierung. Auf Internetseiten mit Text- und Bildinhalten (wie Nachrichtenseiten, Blogbeiträge, Artikel in sozialen Medien) werden Bilder in der Regel an semantisch sinnvollen Positionen im Textfluss platziert. Wir nutzen dies um ein Framework vorzuschlagen, in dem relevante Bilder ausgesucht werden und mit den passenden Abschnitten eines Textes assoziiert werden. • Bildunterschriften. Bilder, die als Teil von multimodalen Inhalten zur Verbesserung der Lesbarkeit von Texten dienen, haben typischerweise Bildunterschriften, die zum Kontext des umgebenden Texts passen. Wir schlagen vor, den Kontext beim automatischen Generieren von Bildunterschriften ebenfalls einzubeziehen. Üblicherweise werden hierfür die Bilder allein analysiert. Wir stellen die kontextbezogene Bildunterschriftengenerierung vor. Unsere vielversprechenden Beobachtungen und Ergebnisse eröffnen interessante Möglichkeiten für weitergehende Forschung zur computergestützten Erfassung des Zusammenspiels von Text- und Bildinhalten

Extracting location and creator-related information from Wikipedia-based information-rich taxonomy for ConceptNet expansion

Our research goal is to generate new assertions suitable for introduction to theJapanese part of the ConceptNet common sense knowledge ontology. In thispaper we present a method for extracting IsA assertions (hyponymy relations),AtLocation assertions (informing of the location of an object or place), Located-Near assertions (informing of neighboring locations) and CreatedBy assertions(informing of the creator of an object) automatically from Japanese WikipediaXML dump les. We use the Hyponymy extraction tool v1.0, which analyzesde nition, category and hierarchy structures of Wikipedia articles to extractIsA assertions and produce an information-rich taxonomy. From this taxon-omy we extract additional information, in this case AtLocation, LocatedNearand CreatedBy types of assertions, using our original method. The presentedexperiments prove that we achieved our research goal on a large scale: bothmethods produce satisfactory results, and we were able to acquire 5,866,680 IsAassertions with 96.0% reliability, 131,760 AtLocation assertion pairs with 93.5%reliability, 6,217 LocatedNear assertion pairs with 98.5% reliability and 270,230CreatedBy assertion pairs with 78.5% reliability. Our method surpassed thebaseline system in terms of both precision and the number of acquired asser-tions

Automatic text summarisation using linguistic knowledge-based semantics

Text summarisation is reducing a text document to a short substitute summary. Since the commencement of the field, almost all summarisation research works implemented to this date involve identification and extraction of the most important document/cluster segments, called extraction. This typically involves scoring each document sentence according to a composite scoring function consisting of surface level and semantic features. Enabling machines to analyse text features and understand their meaning potentially requires both text semantic analysis and equipping computers with an external semantic knowledge. This thesis addresses extractive text summarisation by proposing a number of semantic and knowledge-based approaches. The work combines the high-quality semantic information in WordNet, the crowdsourced encyclopaedic knowledge in Wikipedia, and the manually crafted categorial variation in CatVar, to improve the summary quality. Such improvements are accomplished through sentence level morphological analysis and the incorporation of Wikipedia-based named-entity semantic relatedness while using heuristic algorithms. The study also investigates how sentence-level semantic analysis based on semantic role labelling (SRL), leveraged with a background world knowledge, influences sentence textual similarity and text summarisation. The proposed sentence similarity and summarisation methods were evaluated on standard publicly available datasets such as the Microsoft Research Paraphrase Corpus (MSRPC), TREC-9 Question Variants, and the Document Understanding Conference 2002, 2005, 2006 (DUC 2002, DUC 2005, DUC 2006) Corpora. The project also uses Recall-Oriented Understudy for Gisting Evaluation (ROUGE) for the quantitative assessment of the proposed summarisers’ performances. Results of our systems showed their effectiveness as compared to related state-of-the-art summarisation methods and baselines. Of the proposed summarisers, the SRL Wikipedia-based system demonstrated the best performance