Data triangulation in a user evaluation of the sealife semantic web browsers

There is a need for greater attention to triangulation of data in user-centred evaluation of Semantic Web Browsers. This paper discusses triangulation of data gathered during development of a novel framework for user-centred evaluation of Semantic Web Browsers. The data was triangulated from three sources: quantitative data from web server logs and questionnaire results, and qualitative data from semi-structured interviews. This paper shows how triangulation was essential in validation and completeness of the results, and was indispensable in ensuring accurate interpretation of the results in determining user satisfaction

A user-centred evaluation framework for the Sealife semantic web browsers

BACKGROUND: Semantically-enriched browsing has enhanced the browsing experience by providing contextualized dynamically generated Web content, and quicker access to searched-for information. However, adoption of Semantic Web technologies is limited and user perception from the non-IT domain sceptical. Furthermore, little attention has been given to evaluating semantic browsers with real users to demonstrate the enhancements and obtain valuable feedback. The Sealife project investigates semantic browsing and its application to the life science domain. Sealife's main objective is to develop the notion of context-based information integration by extending three existing Semantic Web browsers (SWBs) to link the existing Web to the eScience infrastructure. METHODS: This paper describes a user-centred evaluation framework that was developed to evaluate the Sealife SWBs that elicited feedback on users' perceptions on ease of use and information findability. Three sources of data: i) web server logs; ii) user questionnaires; and iii) semi-structured interviews were analysed and comparisons made between each browser and a control system. RESULTS: It was found that the evaluation framework used successfully elicited users' perceptions of the three distinct SWBs. The results indicate that the browser with the most mature and polished interface was rated higher for usability, and semantic links were used by the users of all three browsers. CONCLUSION: Confirmation or contradiction of our original hypotheses with relation to SWBs is detailed along with observations of implementation issues

User-guided Disambiguation for Semantic-enriched Search

In this thesis, we develop, implement and evaluate a strategy for user-guided disambiguation for semantic-enriched search. We use the prototype system Capisco that combines some features of traditional text-based search engines and semantic search engines. Unlike many semantic search approaches that require formulating queries in quite complex semantic languages, the Capisco system is a keyword-based search engine. In this work, we focus on the user-guided disambiguation which helps to capture the users’ information needs. The user-guided disambiguation developed in this thesis explores the semantic meanings for a user-provided keyword and displays these to the users for selection. We explored three different orderings in which the possible keyword meanings may be shown. Using our newly-developed disambiguation interface, the user selects one or several of these meanings from the list offered. The selected semantic meanings are then transferred to the search engine to identify matching documents. This thesis provides an analysis of related work on search interfaces in semantic search, and in methods for semantic disambiguation. We developed the concept of user-guided disambiguation and implemented this concept in a prototype that explored three interface variations. We executed a user study on the three interface alternatives. Finally, we discuss the insights gained during this project, compare our approach to existing literature, and explore possible future work

SEMANTIC DATA CLOUDING OVER THE WEBS

Very often, for business or personal needs, users require to retrieve, in a very fast way, all the available relevant information about a focused target entity, in order to take decisions, organize business work, plan future actions. To answer this kind of \u201centity\u201d- driven user needs, a huge multiplicity of web resources is actually available, coming from the Social Web and related user-centered services (e.g., news publishing, social networks, microblogging systems), from the Semantic Web and related ontologies and knowledge repositories, and from the conventional Web of Documents. The Ph.D. thesis is devoted to define the notion of in-cloud and a semantic clouding approach for the construction of in-clouds that works over the Social Web, the Semantic Web, and the Web of Documents. in-clouds are built for a target entity of interest to organize all relevant web resources, modeled as web data items, into a graph, on the basis of their level of prominence and reciprocal closeness. Prominence captures the importance of a web resource within the in-cloud, by distinguishing, also in a visual way \u201ca la tagcloud\u201d, how much relevant web resources are with respect to the target entity. The level of closeness between web resources is evaluated using matching and clustering techniques, with the goal of determining how similar web resources are to each other and with respect to the target entity

User-Oriented Evaluation Methods for Interactive Web Search Interfaces

Although significant efforts have been devoted to the study and evaluation of information retrieval systems from an algorithmic perspective, far less work has been per-formed on the evaluation of these systems from the user’s perspective. This is certainly the case for Web informa-tion retrieval, where the major search engines continue to utilise interfaces that have not changed substantially since their introduction. One of the challenges in developing new Web search interfaces is the evaluation of these systems in comparison to one another, as well as in comparison to the popular Web search engines. In this paper, we highlight some of the methods used in the literature for evaluating Web search systems, and present a summary of the methods that we have found to be effective in dealing with the chal-lenges of evaluating intelligent and interactive Web search interfaces. 1

Metadata visualization of scholarly search results

Studies of online search behaviour have found that searchers often face difficulties formulating queries and exploring the search results sets. These shortcomings may be especially problematic in digital libraries since library searchers employ a wide variety of information-seeking methods (with varying degrees of support), and the corpus to be searched is often more complex than simple textual information. To address these problems, an interactive Web-based library search interface is presented, which has been designed to support strategic retrieval behaviour of library searchers. This system takes advantage of the rich metadata associated with academic documents and employs information visualization techniques to provide searchers with additional information-seeking tools. These tools are designed to facilitate visual and interactive query refinement, search results exploration, and citation navigation. User evaluations illustrate the potential benefits of the design choices in comparison to a list-based digital library search interface

Word-sense disambiguation in biomedical ontologies

With the ever increase in biomedical literature, text-mining has emerged as an important technology to support bio-curation and search. Word sense disambiguation (WSD), the correct identification of terms in text in the light of ambiguity, is an important problem in text-mining. Since the late 1940s many approaches based on supervised (decision trees, naive Bayes, neural networks, support vector machines) and unsupervised machine learning (context-clustering, word-clustering, co-occurrence graphs) have been developed. Knowledge-based methods that make use of the WordNet computational lexicon have also been developed. But only few make use of ontologies, i.e. hierarchical controlled vocabularies, to solve the problem and none exploit inference over ontologies and the use of metadata from publications. This thesis addresses the WSD problem in biomedical ontologies by suggesting different approaches for word sense disambiguation that use ontologies and metadata. The "Closest Sense" method assumes that the ontology defines multiple senses of the term; it computes the shortest path of co-occurring terms in the document to one of these senses. The "Term Cooc" method defines a log-odds ratio for co-occurring terms including inferred co-occurrences. The "MetaData" approach trains a classifier on metadata; it does not require any ontology, but requires training data, which the other methods do not. These approaches are compared to each other when applied to a manually curated training corpus of 2600 documents for seven ambiguous terms from the Gene Ontology and MeSH. All approaches over all conditions achieve 80% success rate on average. The MetaData approach performs best with 96%, when trained on high-quality data. Its performance deteriorates as quality of the training data decreases. The Term Cooc approach performs better on Gene Ontology (92% success) than on MeSH (73% success) as MeSH is not a strict is-a/part-of, but rather a loose is-related-to hierarchy. The Closest Sense approach achieves on average 80% success rate. Furthermore, the thesis showcases applications ranging from ontology design to semantic search where WSD is important

Visualisation des résultats de recherche classifiés en contexte de recherche d’information exploratoire : une évaluation d’utilisabilité

La recherche d’information exploratoire sur le Web présente des défis cognitifs en termes de stratégies cognitives et de tactiques de recherche. Le modèle « question-réponse » des moteurs de recherche actuels est inadéquat pour faciliter les stratégies de recherche d’information exploratoire, assimilables aux stratégies cognitives de l’apprentissage. La visualisation des résultats de recherche est un dispositif qui possède des propriétés graphiques et interactives pertinentes pour le traitement de l’information et l’utilisation de la mémoire et, plus largement de la cognition humaine. Plusieurs recherches ont été menées dans ce contexte de recherche d’information exploratoire, mais aucune n’a distinctement isolé le facteur graphique et interactif de la « visualisation » au sein de son évaluation. L’objectif principal de cette thèse est de vérifier si la visualisation des résultats en contexte de recherche d’information exploratoire témoigne des avantages cognitifs et interactifs pressentis selon ses présupposés théoriques. Pour décrire et déterminer la valeur ajoutée de la visualisation des résultats de recherche dans un contexte de recherche d’information exploratoire sur le Web, cette recherche propose de mesurer son utilisabilité. En la comparant selon les mêmes critères et indicateurs à une interface homologue textuelle, nous postulons que l’interface visuelle atteindra une efficacité, efficience et satisfaction supérieure à l’interface textuelle, dans un contexte de recherche d’information exploratoire. Les mesures objectives de l’efficacité et de l’efficience reposent principalement sur l’analyse des traces de l’interaction des utilisateurs, leur nombre et leur durée. Les mesures subjectives attestant de la satisfaction procurée par l’usage du système dans ce contexte repose sur la perception des utilisateurs par rapport à des critères de perception de la facilité d’utilisation et de l’utilité de l’interface testée et par rapport à des questions plus large sur l’expérience de recherche vécue. Un questionnaire et un entretien ont été passés auprès de chacun des vingt-trois répondants. Leur session de recherche a aussi été enregistré par un logiciel de capture vidéo d’écran. Sur les données des vingt-trois utilisateurs divisés en deux groupes, l’analyse statistique a révélé de faibles différences significatives entre les deux interfaces. Selon les mesures effectuées, l’interface textuelle s’est révélée plus efficace en terme de rappel et de pertinence ; et plus efficiente pour les durées de la recherche d’information. Sur le plan de la satisfaction, les interfaces ont été appréciées toutes deux posivitivement, ne permettant pas de les distinguer pour la grande majorité des métriques. Par contre, au niveau du comportement interactif, des différences notables ont montré que les utilisateurs de l’interface visuelle ont réalisé davantage d’interactions de type exploratoire, et ont procédé à une collecte sélective des résultats de recherche. L’analyse statistique et de contenu sur le critère de l’expérience vécue a permis de démontrer que la visualisation offre l’occasion à l’utilisateur de s’engager davantage dans le processus de recherche d’information en raison de l’impact positif de l’esthétique de l’interface visuelle. De plus, la fonctionnalité de classification a été perçue de manière ambivalente, divisant les candidats peu importe l’interface testée. Enfin, l’analyse des verbatims des « visuelle » a permis d’identifier le besoin de fonctionnalités de rétroaction de l’utilisateur afin de pouvoir communiquer le besoin d’information ou sa pondération des résultats ou des classes, grâce à des modalités interactives de manipulation directe des classes sur un espace graphique.Conducting exploratory searches on the web presents a number of cognitive difficulties as regards search strategies and tactics. The “question-response” model used by the available search engines does not respond adequately to exploratory searches, which are akin to cognitive learning strategies. Visualising search results involves graphic and interactive properties for presenting information that are pertinent for processing and using information, as well as for remembering and, more broadly, for human cognition. Many studies have been conducted in the area of exploratory searches, but none have focussed specifically on the graphic and interactive features of visualisation in their analysis. The principal objective of this thesis is to confirm whether the visualisation of results in the context of exploratory searches offers the cognitive and interactive advantages predicted by conjectural theory. In order to describe and to determine the added value of visualising search results in the context of exploratory web searches, the study proposes to measure its usability. By comparing it to a parallel text interface, using the same criteria and indicators, the likelihood of better efficiency, efficacy, and satisfaction when using a visual interface can be established. The objective measures of efficiency and efficacy are based mainly on the analysis of user interactions, including the number of these interactions and the time they take. Subjective measures of satisfaction in using the system in this context are based on user perception regarding ease of use and the usefulness of the interface tested, and on broader questions concerning the experience of using the search interface. These data were obtained using a questionnaire and a discussion with each participant. Statistical analysis of the data from twenty-three participants divided into two groups showed slightly significant differences between the two interfaces. Analysis of the metrics used showed that the textual interface is more efficient in terms of recall and pertinence, and more efficacious concerning the time needed to search for information. Regarding user satisfaction, both interfaces were seen positively, so that no differences emerged for the great majority of metrics used. However, as regards interactive behaviour, notable differences emerged. Participants using the visual interface had more exploratory interaction, and went on to select and collect pertinent search results. Statistical and content analysis of the experience itself showed that visualisation invites the user to become more involved in the search process, because of the positive effect of a pleasing visual interface. In addition, the classification function was perceived as ambivalent, dividing the participants no matter which interface was used. Finally, analysis of the verbatim reports of participants classed as “visual” indicated the need for a user feedback mechanism in order to communicate information needs or for weighting results or classes, using the interactive function for manipulating classes within a geographic space