Evaluating the features of Museum Websites (The Bologna Report)

MiLE (Milano – Lugano Evaluation Method) is an innovative method for evaluating the quality and usability of hypermedia applications. This paper focuses upon the specific “module” of MiLE concerning cultural heritage applications, synthesizing the results of research carried on by a group of seven museum experts of Bologna (Italy), with the joint coordination of IBC (Institute for the Cultural Heritage of the Emilia Romagna Region) and Politecnico di Milano. The “Bologna group” is composed of different professional figures working in the museum domain: museum curators of artistic, archaeological and historical heritage; museum communication experts; Web sites of cultural institutions’ communication experts. After illustrating the general features of MiLE and the specific features for Cultural Heritage, we will briefly show a few of the results which are to be published in the “Bologna Report”

Towards a Toolset for Intranet Evaluation

Usability is an important component of information systems acceptance. Independent consultants in the assessment of organisational intranets often perform heuristic appraisal, a common method of usability evaluation. However, there are alternative usability models that offer valuable analysis in the evaluation process. Using a government organisation’s intranet as a case study, this paper assesses the value of an independent heuristic-based intranet audit by providing a comparable approach to assessment realisable internally in the organisation using questionnaires. Using a single case study, we empirically apply the Technology Acceptance Model (TAM), together with some heuristic aspects, to provide an alternative tool for intranet usability and acceptance. We provide insight into the usability impact of intranet design changes, and compare the findings of an external usability audit with the approach outlined. An overall toolset for intranet evaluation is proposed as an initial step for further exploration and potential use

Heurísticas de usabilidad para sitios web transaccionales

El presente trabajo de investigación muestra cómo las heurísticas más reconocidas y utilizadas en la actualidad para evaluar el nivel de usabilidad de sitios Web transaccionales presentan falencias e inducen a errores en los resultados, motivo por el cual se determina que las actuales heurísticas ya no son instrumentos apropiados y fiables para evaluar esta característica de calidad en las nuevas y emergentes categorías de aplicaciones de software. Este hecho ha sido el móvil para plantear la siguiente interrogante como tema de investigación: ¿Qué elementos esenciales deberían considerarse en la elaboración de un nuevo conjunto de principios heurísticos de evaluación que permitan medir con mayor precisión el nivel de usabilidad de un sitio Web transaccional? Para solucionar la problemática descrita en párrafos anteriores, se elaboró una propuesta metodológica basada en el análisis de datos de distintos casos de estudio que como resultado presentan un conjunto de principios heurísticos, los cuales han sido desarrollados considerando que pueden ser utilizados por evaluadores con distintos niveles de experticia en evaluaciones de usabilidad. El objetivo de este nuevo conjunto de heurísticas de usabilidad propuesto estuvo orientado a la obtención de principios que sean entendibles, fáciles de utilizar y que a su vez permitan realizar de forma efectiva evaluaciones de usabilidad de Sitios Web Transaccionales. La metodología propuesta está enmarcada básicamente en la mejora del actual instrumento de evaluación propuesto por Jakob Nielsen y otros aspectos que han sido tomados de la literatura. El alcance de la investigación se enmarca dentro del paradigma mixto debido a que la primera fase del trabajo se centró en demostrar la existencia de la problemática planteada a través de un análisis cualitativo de la evidencia obtenida en estudios de casos y en la elaboración de la propuesta heurística de evaluación que cubre aquellos aspectos que no están siendo considerados en la actualidad. Posteriormente, la realización de una evaluación heurística ha permitido validar mediante la aplicación en la práctica, la nueva propuesta desarrollada por el autor del trabajo de investigación; con lo que se consolida un nuevo conocimiento que aporta al bagaje cultural de la ciencia.Tesi

Modeling Complex High Level Interactions in the Process of Visual Mining

Visual Mining refers to the human analytical process that uses visual representations of raw data and makes suitable inferences. During this analytical process, users are engaged in complex cognitive activities such as decision making, problem solving, analytical reasoning and learning. Now a days, users typically use interactive visualization tools, which we call as visual mining support tools (VMSTs), to mediate their interactions with the information present in visual representations of raw data and also to support their complex cognitive activities when performing visual mining. VMSTs have two main components: visual representation and interaction. Even though, these two components are fundamental aspects of VMSTs, the research on visual representation has received the most attention. It is still unclear how to design interactions which can properly support users in performing complex cognitive activities during the visual mining process. Although some fundamental concepts and techniques regarding interaction design have been in place for a while, many established researchers are of the opinion that we do not yet have a generalized, principled, and systematic understanding of interaction components of these VMSTs, and how interactions should be analyzed, designed, and integrated to support complex cognitive activities. Many researchers have recommended that one way to address this problem is through appropriate characterization of interactions in the visual mining process. Models that provide classifications of interactions have indeed been proposed in the visualization research community. While these models are important contributions for the visualization research community, they often characterize interactions at lower levels of human information interaction and high level interactions are not well addressed. In addition, some of these models are not designed to model user activity; rather they are most applicable for representing a system’s response to user activity and not the user activity itself. In this thesis, we address this problem through characterization of the interaction space of visual mining at the appropriate level. Our main contribution in this research is the discovery of a small set of classification criteria which can comprehensively characterize the interaction space of visual mining involving interactions with VMSTs for performing complex cognitive activities. These complex cognitive activities are modeled through visual mining episodes, a coherent set of activities consisting of visual mining strategies (VMSs). Using the classification criteria, VMSs are simply described as combinations of different values of these criteria. By considering all combinations, we can comprehensively cover the interaction space of visual mining. Our VMS interaction space model is unique in identifying the activity tier, a granularity of interactions (high level) which supports performance of complex cognitive activities through interactions with visual information using VMSTs. As further demonstration of the utility of this VMS interaction space model, we describe the formulation of an inspection framework which can provide quantitative measures for the support provided by VMSTs for complex cognitive activities in visual mining. This inspection framework, which has enabled us to produce a new simpler evaluation method for VMSTs in comparison to existing evaluation methods, is based soundly on existing theories and models. Both the VMS interaction space model and the inspection framework present many interesting avenues for further research

An analytical inspection framework for evaluating the search tactics and user profiles supported by information seeking interfaces

Searching is something we do everyday both in digital and physical environments. Whether we are searching for books in a library or information on the web, search is becoming increasingly important. For many years, however, the standard for search in software has been to provide a keyword search box that has, over time, been embellished with query suggestions, Boolean operators, and interactive feedback. More recent research has focused on designing search interfaces that better support exploration and learning. Consequently, the aim of this research has been to develop a framework that can reveal to designers how well their search interfaces support different styles of searching behaviour.The primary contribution of this research has been to develop a usability evaluation method, in the form of a lightweight analytical inspection framework, that can assess both search designs and fully implemented systems. The framework, called Sii, provides three types of analyses: 1) an analysis of the amount of support the different features of a design provide; 2) an analysis of the amount of support provided for 32 known search tactics; and 3) an analysis of the amount of support provided for 16 different searcher profiles, such as those who are finding, browsing, exploring, and learning. The design of the framework was validated by six independent judges, and the results were positively correlated against the results of empirical user studies. Further, early investigations showed that Sii has a learning curve that begins at around one and a half hours, and, when using identical analysis results, different evaluators produce similar design revisions.For Search experts, building interfaces for their systems, Sii provides a Human-Computer Interaction evaluation method that addresses searcher needs rather than system optimisation. For Human-Computer Interaction experts, designing novel interfaces that provide search functions, Sii provides the opportunity to assess designs using the knowledge and theories generated by the Information Seeking community. While the research reported here is under controlled environments, future work is planned that will investigate the use of Sii by independent practitioners on their own projects

On the advantages of a systematic inspection for evaluating hypermedia usability

It is indubitable that usability inspection of complex hypermedia is still an “art, ” in the sense that a great deal is left to the skills, experience, and ability of the inspectors. Training inspectors is difficult and often quite expensive. The Systematic Usability Evaluation (SUE) inspection technique has been proposed to help usability inspectors share and transfer their evaluation know-how, to simplify the hypermedia inspection process for newcomers, and to achieve more effective and efficient evaluation results. SUE inspection is based on the use of evaluation patterns, called abstract tasks, which precisely describe the activities to be performed by evaluators during inspection. This article highlights the advantages of this inspection technique by presenting its empirical validation through a controlled experiment. Two groups of novice inspectors were asked to evaluate a commercial hypermedia CD-ROM by applying the SUE inspection or traditional heuristic evaluation. The comparison was based on three major dimensions: effectiveness, efficiency, and satisfaction. Results indicate a clear advantage of the SUE inspection over the traditional inspection on all dimensions, demonstrating that abstract tasks are efficient tools to drive the evaluator’s performance. The authors are immensely grateful to Prof. Rex Hartson, from Virginia Tech, for his valuable suggestions. The authors also thank Francesca Alonzo and Alessandra Di Silvestro, from the Hypermedia Open Center of Polytechnic of Milan, for the help offered during the experiment data coding. The support of the EC grant FAIRWIS project IST-1999-12641 and of MURST COFIN 2000 is acknowledged

Der Einfluss von Inhaltsgestaltung und Lernereigenschaften auf Navigation und Wissenserwerb in hypermedialen Lernumgebungen

Educational software increasingly finds its way into classroom learning. The question arises whether the currently available products ensure effectiveness and efficiency of learning. So far investigations on the learning process when using hypermedia environments and moreover the factors impacting on the software use are scarce. The presented doctoral thesis, set in a realistic classroom scenario and presenting a complex learner-controlled hypermedia environment, aims to brigde this gap. Four evaluation variants of a CD-ROM on cell biology offering different didactical and graphical designs but the same content were produced. Specifically, the impacts of 3D-models, close-up views, static picture as well as animation design were investigated. About 700 students of 15 schools and 3 universities in 5 different German ôBundeslaenderö participated in the study. The knowledge acquisition and navigation behaviour of the examinees was documented with encoded log-files and correlated with the individualÆs learner characteristics such as learning style, prior domain knowledge and visual spatial ability as well as the design features of the software variants. Individual as well as group learning was analysed. Principle component analysis (PCA) was applied to get main usage factors that served as dependent variables in the subsequently calculated statistical tests. The dissertation suggests various influences of learner characteristics and content design on the learning process thereby contributing to the design of learning software that is well adapted to usersÆ needs. The studyÆs results are discussed in light of an extended cognitive load theory for hypermedia learning environments.Neue Medien kommen immer öfter auch im Klassenzimmer zum Einsatz. Die Frage stellt sich daher, ob die derzeit erhältlichen Produkte die Effektivität und Effizienz des Lernens gewährleisten. Untersuchungen, die den Lernprozess bei der Nutzung hypermedialer Lernumgebungen und darüber hinaus die die Softwarenutzung beeinflussenden Faktoren beleuchten, sind bislang rar. Die vorliegende Doktorarbeit, unter Verwendung einer Nutzer-kontrollierten hypermedialen Lernumgebung im Klassenraum durchgeführt, und damit die realistische Lernsituation simulierend, zielt darauf ab, diese Lücke zu schließen. Vier Evaluationsvarianten einer CD - ROM zum Thema Zellbiologie verschiedenen didaktischen und graphischen Designs aber gleichen Inhalts wurden produziert. Insbesondere wurden der Einfluss von 3D-Modellen, Detaildarstellungen, des Designs statischer Bilder sowie von Animationen untersucht. Etwa 700 Schüler/innen und Studierende aus 15 Schulen und 3 Universitäten in 5 verschiedenen deutschen Bundesländern nahmen an der Studie teil. Navigation und Prozess des Wissenserwerbs der Teilnehmer wurde mit codierten Logfiles erfasst und mit individuellen Eigenschaften wie Lernstil, fachlichem Vorwissen und räumlichem Vorstellungsvermögen sowie den Designmerkmalen der Softwarevarianten korreliert. Sowohl Einzel- als auch Gruppenlernen wurde untersucht. Mittels einer Faktorenanalyse (PCA) wurden Hauptnutzungsfaktoren ermittelt, die in den nachfolgend durchgeführten statistischen Tests als abhängige Variablen fungierten. Die Studie zeigt verschiedene Einflüsse der Nutzereigenschaften und des inhaltlichen Designs auf den Lernprozess und unterstützt dadurch die Entwicklung von Softwareprodukten, die den Bedürfnissen der Nutzer ausreichend Rechnung tragen. Die Ergebnisse werden im Zusammenhang einer erweiterten kognitiven Beanspruchungstheorie („Cognitive Load Theory“) für hypermediale Lernumgebungen diskutiert

DRAFT SUBMITTED TO INTERNATIONAL JOURNAL OF HUMAN COMPUTER INTERACTION On the Advantages of a Systematic Inspection for Evaluating Hypermedia Usability

It is indubitable that usability inspection of complex hypermedia is still an "art", in the sense that a great deal is left to the skills, experience, and ability of the inspectors. Training inspectors is difficult and often quite expensive. The SUE inspection technique has been proposed to help usability inspectors share and transfer their evaluation know-how, to simplify the hypermedia inspection process for newcomers, and to achieve more effective and efficient evaluation results. SUE is based on the use of evaluation patterns, called Abstract Tasks, which precisely describe the activities to be performed by evaluators during inspection. This paper highlights the advantages of this inspection technique, by presenting its empirical validation through a controlled experiment. Two groups of novice inspectors have been asked to evaluate a commercial hypermedia CD-ROM applying the SUE inspection or the traditional heuristic technique. The comparison was based on three major dimensions: effectiveness, efficiency, and satisfaction. Results have shown a clear advantage of the SUE inspection over the traditional inspection on all dimensions, demonstrating that Abstract Tasks are efficient tools to drive the evaluator's performance. 1