End-User Evaluations of Semantic Web Technologies

Stanford University's Knowledge Systems Laboratory (KSL) is working in partnership with Battelle Memorial Institute and IBM Watson Research Center to develop a suite of technologies for information extraction, knowledge representation & reasoning, and human-information interaction, in unison entitled 'Knowledge Associates for Novel Intelligence' (KANI). We have developed an integrated analytic environment composed of a collection of analyst associates, software components that aid the user at different stages of the information analysis process. An important part of our participatory design process has been to ensure our technologies and designs are tightly integrate with the needs and requirements of our end users, To this end, we perform a sequence of evaluations towards the end of the development process that ensure the technologies are both functional and usable. This paper reports on that process

Embedding accessibility and usability: considerations for e-learning research and development projects

This paper makes the case that if e‐learning research and development projects are to be successfully adopted in real‐world teaching and learning contexts, then they must effectively address accessibility and usability issues; and that these need to be integrated throughout the project. As such, accessibility and usability issues need to be made explicit in project documentation, along with allocation of appropriate resources and time. We argue that accessibility and usability are intrinsically inter‐linked. An integrated accessibility and usability evaluation methodology that we have developed is presented and discussed. The paper draws on a series of mini‐case studies from e‐learning projects undertaken over the past 10 years at the Open University

Socialising around media. Improving the second screen experience through semantic analysis, context awareness and dynamic communities

SAM is a social media platform that enhances the experience of watching video content in a conventional living room setting, with a service that lets the viewer use a second screen (such as a smart phone) to interact with content, context and communities related to the main video content. This article describes three key functionalities used in the SAM platform in order to create an advanced interactive and social second screen experience for users: semantic analysis, context awareness and dynamic communities. Both dataset-based and end user evaluations of system functionalities are reported in order to determine the effectiveness and efficiency of the components directly involved and the platform as a whole

Closed system approach to cell expansion

This poster describes the development of a novel closed system approach to cell culture expansion in shaker flasks. An insert with microporous membranes was created and combined with a cap closure with integral tubing attached to the cap for the aseptic input of media and inoculate and for the aseptic output of samples and finished product. The pH of the solution was measured as a function of carbon dioxide concentration and compared with traditional shaker flasks. Cell culture doubling times, cell viability, and total cell counts were measured for 500 mL, 1,000 mL, and 3,000 mL systems. Additive manufacturing was also used to speed up the evaluation of the technology. Sample caps were 3D printed for end user evaluations of the early prototypes. Closed system processing is now a viable option for cell culture expansions

An examination of interactions between computer support staff and end users

For those employed as information technology support professionals, the message they are receiving is that communication skills are as important as any credential or certification they may possess in terms of staying competitive in the current job market. The goal of this study was to determine the importance of communication skills in interactions between information technology support staff and the end users they support in relation to factors such as stereotypes and expectations. The results of this study suggest that positive end user evaluations of these communication interactions do not rely solely upon the communication skills of the technician, but instead may also be significantly influenced by other equally important factors. However, an argument can certainly be made for technicians to develop communication skills, as these skills should be positively valanced by end users and as a result, should aid in reducing end user uncertainty about future interactions with technicians

XIP Dashboard: visual analytics from automated rhetorical parsing of scientific metadiscourse

A key competency that we seek to build in learners is a critical mind, i.e. ability to engage with the ideas in the literature, and to identify when significant claims are being made in articles. The ability to decode such moves in texts is essential, as is the ability to make such moves in one’s own writing. Computational techniques for extracting them are becoming available, using Natural Language Processing (NLP) tuned to recognize the rhetorical signals that authors use when making a significant scholarly move. After reviewing related NLP work, we introduce the Xerox Incremental Parser (XIP), note previous work to render its output, and then motivate the design of the XIP Dashboard, a set of visual analytics modules built on XIP output, using the LAK/EDM open dataset as a test corpus. We report preliminary user reactions to a paper prototype of such a novel dashboard, describe the visualizations implemented to date, and present user scenarios for learners, educators and researchers. We conclude with a summary of ongoing design refinements, potential platform integrations, and questions that need to be investigated through end-user evaluations

Underdogs and superheroes: Designing for new players in public space

We are exploring methods for participatory and public involvement of new 'players' in the design space. Underdogs & Superheroes involves a game-based methodology – a series of creative activities or games – in order to engage people experientially, creatively, and personally throughout the design process. We have found that games help engage users’ imaginations by representing reality without limiting expectations to what's possible here and now; engaging experiential and personal perspectives (the 'whole' person); and opening the creative process to hands-on user participation through low/no-tech materials and a widely-understood approach. The methods are currently being applied in the project Underdogs & Superheroes, which aims to evolve technological interventions for personal and community presence in local public spaces

Evaluating accessible adaptable e-learning

Two different approaches can increase the accessibility of digital educational materials: content that has been built with the widest possible set of users in mind (universal design), or content that has been designed in such a way that it can be personalised to individual user needs and preferences (personalised design). This paper outlines a number of approaches that could be used to evaluate the provision of learning materials that have been adapted to or chosen for individual learners. A number of different perspectives are considered in this paper: a learner's perspective, the perspective of the tutor or teacher, and an institutional perspective. A number of complementary methodologies are presented. It is argued that the evaluation of a system that provides personalised learning content is a challenging activity that necessitates the application of multiple methods to effectively understand the underlying costs and benefits of providing personalised learning materials

Toward automatic comparison of visualization techniques: Application to graph visualization

Many end-user evaluations of data visualization techniques have been run during the last decades. Their results are cornerstones to build efficient visualization systems. However, designing such an evaluation is always complex and time-consuming and may end in a lack of statistical evidence and reproducibility. We believe that modern and efficient computer vision techniques, such as deep convolutional neural networks (CNNs), may help visualization researchers to build and/or adjust their evaluation hypothesis. The basis of our idea is to train machine learning models on several visualization techniques to solve a specific task. Our assumption is that it is possible to compare the efficiency of visualization techniques based on the performance of their corresponding model. As current machine learning models are not able to strictly reflect human capabilities, including their imperfections, such results should be interpreted with caution. However, we think that using machine learning-based pre-evaluation, as a pre-process of standard user evaluations, should help researchers to perform a more exhaustive study of their design space. Thus, it should improve their final user evaluation by providing it better test cases. In this paper, we present the results of two experiments we have conducted to assess how correlated the performance of users and computer vision techniques can be. That study compares two mainstream graph visualization techniques: node-link (\NL) and adjacency-matrix (\MD) diagrams. Using two well-known deep convolutional neural networks, we partially reproduced user evaluations from Ghoniem \textit{et al.} and from Okoe \textit{et al.}. These experiments showed that some user evaluation results can be reproduced automatically.Comment: 35 pages, 6 figures, 4 table

MyCare Card Development: Portable GUI Framework for the Personal Electronic Health Record Device

In most emergency situations, health professionals rely on patients to provide information about their medical history. However, in some cases patients might not be able to communicate this information, and in most countries an online integrated patient record system has not been adopted yet. Therefore, in order to address this issue the ongoing project MyCare Card (MyC2, www.myc2.org) has been established. The aim of this project is to design, implement, and evaluate a prototype patient held electronic health record device. Due to the wide range of user requirements, the device, its communication interface, and its software have to be compatible with many common platforms and operating systems. Thus, this paper is addressing one of the software compatibility matters-the cross-platform GUI implementation. It introduces a portable object-oriented GUI framework, suitable for a declarative layout definition, components customization, and fine model-view code separation. It also rationalizes the hardware and software solutions selected for this project implementation