Usability Evaluation in Virtual Environments: Classification and Comparison of Methods

Virtual environments (VEs) are a relatively new type of human-computer interface in which users perceive and act in a three-dimensional world. The designers of such systems cannot rely solely on design guidelines for traditional two-dimensional interfaces, so usability evaluation is crucial for VEs. We present an overview of VE usability evaluation. First, we discuss some of the issues that differentiate VE usability evaluation from evaluation of traditional user interfaces such as GUIs. We also present a review of VE evaluation methods currently in use, and discuss a simple classification space for VE usability evaluation methods. This classification space provides a structured means for comparing evaluation methods according to three key characteristics: involvement of representative users, context of evaluation, and types of results produced. To illustrate these concepts, we compare two existing evaluation approaches: testbed evaluation [Bowman, Johnson, & Hodges, 1999], and sequential evaluation [Gabbard, Hix, & Swan, 1999]. We conclude by presenting novel ways to effectively link these two approaches to VE usability evaluation

Evaluating Software Architectures: Development Stability and Evolution

We survey seminal work on software architecture evaluationmethods. We then look at an emerging class of methodsthat explicates evaluating software architectures forstability and evolution. We define architectural stabilityand formulate the problem of evaluating software architecturesfor stability and evolution. We draw the attention onthe use of Architectures Description Languages (ADLs) forsupporting the evaluation of software architectures in generaland for architectural stability in specific

Towards an evaluation framework for medical web applications

Copyright @ 2013 EMCIS.The main aim of this study is to review and analyse various evaluation frameworks used to assess the operational effectiveness of various Information Technology (IT) processes/applications and identify their strengths in order to form a new holistic framework for economic evaluation of web applications. This research aims to address the need for a new holistic evaluation framework for the purpose of the evaluation of the medical web applications. Over the last decade more and more, companies used accountancy techniques such as the frameworks analysed in this research. This new holistic framework that was developed will include also steps regarding the indirect and intangible costs and benefits identification and their incorporation in the evaluation process. Moreover the new emerging market of the medical websites and the embedded on them web applications requires also a new evaluation framework that will provide accurate results in the estimation of the efficiency of an investment on them. The paper first presents an introduction about why economic evaluation is important when evaluating the Information Technology in organizations. Various studies are reviewed, which highlight the ever increasing importance of integrating economic evaluation processes, such as Cost Benefit Analysis (CBA) and Return on Investment (ROI), into systems and processes of organizations and economic organizations, and analyse the factors that govern their role. Finally, the evaluation frameworks and methods that are found in these studies should be used as a part of a proactive systematic action plan that the organizations could use to avoid budget reduction due to incorrect planning. The next part of this study includes a comprehensive presentation and review of past frameworks used to evaluate Information Technology. The frameworks that are reviewed are the Framework for evaluation of information systems, the Information Technology Adoption Model (ITAM), the Total Evaluation and Acceptance Methodology (TEAM) framework, the ROI Process Model and HOT-fit evaluation framework for Health Information Systems

Error by design: Methods for predicting device usability

This paper introduces the idea of predicting ‘designer error’ by evaluating devices using Human Error Identification (HEI) techniques. This is demonstrated using Systematic Human Error Reduction and Prediction Approach (SHERPA) and Task Analysis For Error Identification (TAFEI) to evaluate a vending machine. Appraisal criteria which rely upon user opinion, face validity and utilisation are questioned. Instead a quantitative approach, based upon signal detection theory, is recommended. The performance of people using SHERPA and TAFEI are compared with heuristic judgement and each other. The results of these studies show that both SHERPA and TAFEI are better at predicting errors than the heuristic technique. The performance of SHERPA and TAFEI are comparable, giving some confidence in the use of these approaches. It is suggested that using HEI techniques as part of the design and evaluation process could help to make devices easier to use

Opportunistic linked data querying through approximate membership metadata

Between URI dereferencing and the SPARQL protocol lies a largely unexplored axis of possible interfaces to Linked Data, each with its own combination of trade-offs. One of these interfaces is Triple Pattern Fragments, which allows clients to execute SPARQL queries against low-cost servers, at the cost of higher bandwidth. Increasing a client's efficiency means lowering the number of requests, which can among others be achieved through additional metadata in responses. We noted that typical SPARQL query evaluations against Triple Pattern Fragments require a significant portion of membership subqueries, which check the presence of a specific triple, rather than a variable pattern. This paper studies the impact of providing approximate membership functions, i.e., Bloom filters and Golomb-coded sets, as extra metadata. In addition to reducing HTTP requests, such functions allow to achieve full result recall earlier when temporarily allowing lower precision. Half of the tested queries from a WatDiv benchmark test set could be executed with up to a third fewer HTTP requests with only marginally higher server cost. Query times, however, did not improve, likely due to slower metadata generation and transfer. This indicates that approximate membership functions can partly improve the client-side query process with minimal impact on the server and its interface

Issues in Evaluating Health Department Web-Based Data Query Systems: Working Papers

Compiles papers on conceptual and methodological topics to consider in evaluating state health department systems that provide aggregate data online, such as taxonomy, logic models, indicators, and design. Includes surveys and examples of evaluations