Layered evaluation of interactive adaptive systems : framework and formative methods

Peer reviewedPostprin

Records management capacity and compliance toolkits : a critical assessment.

This article seeks to present the results of a project that critically evaluated a series of toolkits for assessing records management capacity and/or compliance. These toolkits have been developed in different countries and sectors within the context of the e-environment and provide evidence of good corporate and information governance. Design/methodology/approach - A desk-based investigation of the tools was followed by an electronic Delphi with toolkit developers and performance measurement experts to develop a set of evaluation criteria. Different stakeholders then evaluated the toolkits against the criteria using cognitive walkthroughs and expert heuristic reviews. The results and the research process were reviewed via electronic discussion. Findings - Developed by recognised and highly respected organisations, three of the toolkits are software tools, whilst the fourth is a methodology. They are all underpinned by relevant national/international records management legislation, standards and good practice including, either implicitly or explicitly, ISO 15489. They all have strengths, complementing rather than competing with one another. They enable the involvement of other staff, thereby providing an opportunity for raising awareness of the importance of effective records management. Practical implications - These toolkits are potentially very powerful, flexible and of real value to organisations in managing their records. They can be used for a "quick and dirty" assessment of records management capacity or compliance as well as in-depth analysis. The most important criterion for selecting the appropriate one is to match the toolkit with the scenario. Originality/value - This paper aims to raise awareness of the range and nature of records management toolkits and their potential for varied use in practice to support more effective management of records

Reviewing and extending the five-user assumption: A grounded procedure for interaction evaluation

" © ACM, 2013. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in ACM Transactions on Computer-Human Interaction (TOCHI), {VOL 20, ISS 5, (November 2013)} http://doi.acm.org/10.1145/2506210 "The debate concerning how many participants represents a sufficient number for interaction testing is well-established and long-running, with prominent contributions arguing that five users provide a good benchmark when seeking to discover interaction problems. We argue that adoption of five users in this context is often done with little understanding of the basis for, or implications of, the decision. We present an analysis of relevant research to clarify the meaning of the five-user assumption and to examine the way in which the original research that suggested it has been applied. This includes its blind adoption and application in some studies, and complaints about its inadequacies in others. We argue that the five-user assumption is often misunderstood, not only in the field of Human-Computer Interaction, but also in fields such as medical device design, or in business and information applications. The analysis that we present allows us to define a systematic approach for monitoring the sample discovery likelihood, in formative and summative evaluations, and for gathering information in order to make critical decisions during the interaction testing, while respecting the aim of the evaluation and allotted budget. This approach – which we call the ‘Grounded Procedure’ – is introduced and its value argued.The MATCH programme (EPSRC Grants: EP/F063822/1 EP/G012393/1

Information systems evaluation methodologies

Due to the prevalent use of Information Systems (IS) in modern organisations nowadays, evaluation research in this field is becoming more and more important. In light of this, a set of rigorous methodologies were developed and used by IS researchers and practitioners to evaluate the increasingly complex IS implementation used. Moreover, different types of IS and different focusing perspectives of the evaluation require the selection and use of different evaluation approaches and methodologies. This paper aims to identify, explore, investigate and discuss the various key methodologies that can be used in IS evaluation from different perspectives, namely in nature (e.g. summative vs. formative evaluation) and in strategy (e.g. goal-based, goal-free and criteria-based evaluation). The paper concludes that evaluation methodologies should be selected depending on the nature of the IS and the specific goals and objectives of the evaluation. Nonetheless, it is also proposed that formative criteria-based evaluation and summative criteria-based evaluation are currently among the most and more widely used in IS research. The authors suggest that the combines used of one or more of these approaches can be applied at different stages of the IS life cycle in order to generate more rigorous and reliable evaluation outcomes

User involvement in healthcare technology development and assessment: Structured literature review

Purpose – Medical device users are one of the principal stakeholders of medical device technologies. User involvement in medical device technology development and assessment is central to meet their needs. Design/methodology/approach – A structured review of literature, published from 1980 to 2005 in peer-reviewed journals, was carried out from social science perspective to investigate the practice of user involvement in the development and assessment of medical device technologies. This was followed by qualitative thematic analysis. Findings – It is found that users of medical devices include clinicians, patients, carers and others. Different kinds of medical devices are developed and assessed by user involvement. The user involvement occurs at different stages of the medical device technology lifecycle and the degree of user involvement is in the order of design stage > testing and trials stage > deployment stage > concept stage. Methods most commonly used for capturing users’ perspectives are usability tests, interviews and questionnaire surveys. Research limitations/implications – We did not review the relevant literature published in engineering, medical and nursing fields, which might have been useful. Practical implications – Consideration of the users’ characteristics and the context of medical device use is critical for developing and assessing medical device technologies from users’ perspectives. Originality/value – This study shows that users of medical device technologies are not homogeneous but heterogeneous, in several aspects, and their needs, skills and working environments vary. This is important consideration for incorporating users’ perspectives in medical device technologies. Paper type: Literature review

Evaluating groupware support for software engineering students

Software engineering tasks, during both development and maintenance, typically involve teamwork using computers. Team members rarely work on isolated computers. An underlying assumption of our research is that software engineering teams will work more effectively if adequately supported by network-based groupware technology. Experience of working with groupware and evaluating groupware systems will also give software engineering students a direct appreciation of the requirements of engineering such systems. This research is investigating the provision of such network-based support for software engineering students and the impact these tools have on their groupwork. We will first describe our experiences gained through the introduction of an asynchronous virtual environment ­ SEGWorld to support groupwork during the Software Engineering Group (SEG) project undertaken by all second year undergraduates within the Department of Computer Science. Secondly we will describe our Computer Supported Cooperative Work (CSCW) module which has been introduced into the students' final year of study as a direct result of our experience with SEG, and in particular its role within Software Engineering. Within this CSCW module the students have had the opportunity to evaluate various groupware tools. This has enabled them to take a retrospective view of their experience of SEGWorld and its underlying system, BSCW, one year on. We report our findings for SEG in the form of a discussion of the hypotheses we formulated on how the SEGs would use SEGWorld, and present an initial qualitative assessment of student feedback from the CSCW module

Recent and upcoming BCI progress: overview, analysis, and recommendations

Brain–computer interfaces (BCIs) are finally moving out of the laboratory and beginning to gain acceptance in real-world situations. As BCIs gain attention with broader groups of users, including persons with different disabilities and healthy users, numerous practical questions gain importance. What are the most practical ways to detect and analyze brain activity in field settings? Which devices and applications are most useful for different people? How can we make BCIs more natural and sensitive, and how can BCI technologies improve usability? What are some general trends and issues, such as combining different BCIs or assessing and comparing performance? This book chapter provides an overview of the different sections of this book, providing a summary of how authors address these and other questions. We also present some predictions and recommendations that ensue from our experience from discussing these and other issues with our authors and other researchers and developers within the BCI community. We conclude that, although some directions are hard to predict, the field is definitely growing and changing rapidly, and will continue doing so in the next several years

Making GDPR Usable: A Model to Support Usability Evaluations of Privacy

We introduce a new model for evaluating privacy that builds on the criteria proposed by the EuroPriSe certification scheme by adding usability criteria. Our model is visually represented through a cube, called Usable Privacy Cube (or UP Cube), where each of its three axes of variability captures, respectively: rights of the data subjects, privacy principles, and usable privacy criteria. We slightly reorganize the criteria of EuroPriSe to fit with the UP Cube model, i.e., we show how EuroPriSe can be viewed as a combination of only rights and principles, forming the two axes at the basis of our UP Cube. In this way we also want to bring out two perspectives on privacy: that of the data subjects and, respectively, that of the controllers/processors. We define usable privacy criteria based on usability goals that we have extracted from the whole text of the General Data Protection Regulation. The criteria are designed to produce measurements of the level of usability with which the goals are reached. Precisely, we measure effectiveness, efficiency, and satisfaction, considering both the objective and the perceived usability outcomes, producing measures of accuracy and completeness, of resource utilization (e.g., time, effort, financial), and measures resulting from satisfaction scales. In the long run, the UP Cube is meant to be the model behind a new certification methodology capable of evaluating the usability of privacy, to the benefit of common users. For industries, considering also the usability of privacy would allow for greater business differentiation, beyond GDPR compliance.Comment: 41 pages, 2 figures, 1 table, and appendixe