Assessing the reliability, validity and acceptance of a classification scheme of usability problems (CUP)

Post-print (lokagerð höfunda)The aim of this study was to evaluate the Classification of Usability Problems (CUP) scheme. The goal of CUP is to classify usability problems further to give user interface developers better feedback to improve their understanding of usability problems, help them manage usability maintenance, enable them to find effective fixes for UP, and prevent such problems from reoccurring in the future. First, reliability was evaluated with raters of different levels of expertise and experience in using CUP. Second, acceptability was assessed with a questionnaire. Third, validity was assessed by developers in two field studies. An analytical comparison was also made to three other classification schemes. CUP reliability results indicated that the expertise and experience of raters are critical factors for assessing reliability consistently, especially for the more complex attributes. Validity analysis results showed that tools used by developers must be tailored to their working framework, knowledge and maturity. The acceptability study showed that practitioners are concerned with the effort spent in applying any tool. To understand developers’ work and the implications of this study two theories are presented for understanding and prioritising UP. For applying classification schemes, the implications of this study are that training and context are needed.Peer Reviewe

SCOPE FOR USABILITY TESTS IN IS DEVELOPMENT

Despite being a common, established concept in wide usage, usability tests can vary greatly in goals, techniques and results. A usability test purchased and performed for a specific software product, may result in either minor user interface improvements or radical U-turns in the development. Such variation has been discussed as a problem of the scientific reliability and validity of the testing method. In practice it is more important what ‘kind of data’ one can expect of the selected method than whether it is reliably always the same data. This expectation of information content or ‘scope’ is of importance for evaluators, who select and conduct usability tests for a specific purpose. However, the scope is not explicitly stated or even discussed: Too often the premise is that, because a usability test involves users, it brings the (necessary) user-centeredness to the design i.e. takes socio-technical fundamentals as inherently given. Through a literature review of testing practices and analytical considerations, we search for the scope of a usability test, which could deliberately approach the socio-technical tradition and equally develop both the system and the user organization. A case example represents a possible realization of the extended scope of usability test

A METHOD FOR CLASSIFYING USABILITY FINDINGS TO ENHANCE VALIDATION OF NEW HEURISTICS

Different usability heuristics have been proposed as new application domains arise. Such proposals usually depend on the validation of the new heuristics. However, current validation methods are still biased by subjective comparisons of usability findings. In this paper, we aimed to enhance the process of matching usability finding descriptions and mitigate the bias of such process. To reach our goal, we adopted ontology techniques to extend the User Action Framework for the context of validating new usability heuristics. We tested three hypotheses about the feasibility of our new framework based on a case study with 173 usability findings. These usability findings were retrieved from an online project of a private mobile browser. Our data analysis of supported merging three classification schemes for our framework: User Action Framework, Typical Usability Defects (from ISO) and the heuristics of Nielsen. Finally, we describe a logical process for our method, because some of the contents from the classification schemes are not disjoint

ENFOQUE METODOLÓGICO PARA EL DISEÑO DE INTERFACES DURANTE EL CICLO DE VIDA DE DESARROLLO DE SOFTWARE

RESUMEN ANALÍTICOLa ingeniería de software busca defnir procesos, controlarlos y seguirlos con el fn de construir productos de calidad que garanticen al usuario su satisfacción, generalmente medida en términos de funcionalidad del producto, pero pocas veces evaluada en términos de usabilidad del mismo. Este artículo presenta un modelo para el ciclo de vida de desarrollo de software, basado en la metodología Human-Centered Design (HCD) en el cual son integrados algunos métodos utilizados en el diseño de producto (diseño industrial). Este modelo incluye métodos de usabilidad (efciencia, efcacia y satisfacción) y técnicas para el diseño de interfaces gráfcas de usuario (apreciación y estética). La investigación utilizó una metodología de análisis cualitativo a través del análisis de la experiencia de una PYME de desarrollo de software. Por último, se utilizó el modelo de referencia Capability and Maturity Model Integrated (CMMI) para la validación del modelo propuesto.PALABRAS CLAVES: Usabilidad, Diseño Centrato en el Usuario, Diseño como Proceso, Ciclo de Vida del Producto.METHODOLOGICAL APPROACH FOR INTERFACE DESIGN DURING SOFTWARE DEVELOPMENT LIFE-CYCLEANALYTICAL SUMMARYSoftware engineering looks for defning, controlling and following processes, it aiming to construct high quality products in order to guarantee user satisfaction. Usually, it is measured in functionality terms, but few times measured from the perspective of usability. This paper presents a life cycle model of software development, based on the Human Centered Design (HCD) methodology, which integrates some methods used in product design (industrial design). The model includes usability methods (effciency, effcacy and satisfaction) and techniques for graphic user interfaces (aesthetic and visual perception). The research uses an analytic qualitative methodology approach by the experience of a SME (Small and Medium Enterprises) of software development. Finally, Capability and Maturity Model Integrated (CMMI) was used as reference for validating the model.KEYWORDS: Usability, Human-Centered Design, design as process, product life cycle

Scope of usability tests in IS development

Despite being a common, established concept in wide usage, usability tests can vary greatly in their goals, techniques, and results. A usability test that one purchases and performs for a specific software product may result in either minor user interface improvements or radical U-turns in development. Researchers have discussed such variation as a problem that concerns testing method’s scientific reliability and validity. In practice, what “kind of data” one can expect to obtain from the selected method has more importance than whether one always obtains the same data. This expectation about information content or “scope” has importance for those who select and conduct usability tests for a specific purpose. However, researchers rarely explicitly state or even discuss scope: too often they adopt the premise that, because a usability test involves users, it brings the (necessary) user-centeredness to the design (i.e., takes socio- technical fundamentals as inherently given). We reviewed the literature on testing practices and analytical consideration and searched for the scope of a usability test that could deliberately approach the socio-technical tradition and equally develop both the system and the user organization. A case example represents a possible realization of the extended scope of usability test.</p

Product Information Quality : A sustainability challenge in design and construction

The adverse consequences of building product performance pose sustainability problems for the built environment. Effective approaches to these problems require a clear understanding of building product information and its provision by manufacturers. This is an essential need for sustainable growth in industrialized construction, a system characterized by the expanded role of the manufacturing sector. Furthermore, a sustainable transition to digitalization in the construction industry needs digital interfaces capable of providing the information required for sustainable design and construction. The aim of this research is to contribute to an increased understanding of how building product information can support sustainability in the built environment. To this end, two fundamental aspects have been examined: the quality of information on the sustainability performance of building products and the usability of the digital interfaces providing such information. This research relies on critical realism and adopts a qualitative methodology to analyze and explain the mechanisms of creating and providing product information in four sequential case studies. Systems thinking and process tracing method have been applied to analyze the flow of product information in the construction industry, the operative processes that can support sustainability, and the stakeholders involved. In the first three case studies, the operative process is the diffusion of innovative ventilation products with superior indoor environmental performance. The first case study identifies the problems affecting this process. The second and third case studies, respectively, explore how product information and information exchange on building information modeling (BIM) library platforms can support the process. Influenced by the Grenfell Tower fire in London in 2017, the fourth case identifies the product information problems that can contribute to harmful facade fires threatening sustainability in the built environment. The study examines the capabilities for avoiding the identified problems and explores how an operative process of design, manufacturing, and construction of fire-safe facades can be supported. The findings reveal problems concerning the quality of information on the sustainability performance of products and the methods used by manufacturers for presenting such information. These problems have limited the availability and usability of the information in product databases and BIM object libraries. This defective flow of information affects the design process and can lead to unacceptable consequences such as facade fires. In addition, the inefficient methods of supplying product information have impeded the adoption of innovative products with improved sustainability performance. To address these issues, this research proposes the standardization of product information in collaboration with effective legislation and establishes a framework for evaluating the provision of information on the sustainability performance of building products. The theoretical contributions of this work include five tools: (1) a model for applied critical realism towards sustainability, (2) a matrix for the qualitative analysis of BIM object library platforms, (3) a matrix for evaluating the quality of information and digital interfaces, (4) a model of the functions of the standards on product information, and (5) a conceptual model of product information for sustainable design and construction

Understanding Usability-related Information Security Failures in a Healthcare Context

This research study explores how the nature and type of usability failures impact task performance in a healthcare organization. Healthcare organizations are composed of heterogeneous and disparate information systems intertwined with complex business processes that create many challenges for the users of the system. The manner in which Information Technology systems and products are implemented along with the overlapping intricate tasks the users have pose problems in the area of usability. Usability research primarily focuses on the user interface; therefore, designing a better interface often leaves security in question. When usability failures arise from the incongruence between healthcare task and the technology used in healthcare organizations, the security of information is jeopardized. Hence, the research problem is to understand the nature and types of usability-related security failures and how they can be reduced in a Healthcare Information System. This research used a positivist single case study design with embedded units, to understand the nature and type of usability-related information systems security failures in a Healthcare context. The nature and types of usability failures were identified following a four-step data analysis process that used terms that defined (1) user failures in a large healthcare organization, (2) Task Technology Fit theory, (3) the Confidentiality Integrity and Availability triad of information protection that captured usability-related information system security failures, and (4) by conducting semi-structured interviews with users of the Healthcare Information System capturing and recording their interactions with the usability failure. The captured reported usability-related information system security failures dated back five years within a healthcare organization consisting of a network of 128 medical centers. The evaluation of five years of data and over 8,000 problems reported by healthcare workers allowed this research to identify the misalignment of healthcare task to the technology used, and how the misalignment impacted both information security and user performance. The nature of usability failures were centered on technical controls, however, the cause of the failures was predominately information integrity failures and the unavailability of applications and systems. Usability-related information system security failures are primarily not recognized due to the nature of healthcare task along with the methods healthcare workers use to mitigate such failures by employing workarounds to complete a task. Applying non-technical security controls within the development process provides the clearest path to addressing throughout the organization the captured usability-related information system security failures

Usability-Ergebnisse als Wissensressource in Organisationen

Durch den Prozess der nutzerzentrierten Softwareentwicklung sammeln Organisationen wichtige Erkenntnisse über die Nutzer ihre Produkte, deren Arbeitsaufgaben und über die Nutzungskontexte, in dem diese sie anwenden. Diese Arbeit untersucht, wie derartige Usability-Ergebnisse in einer Organisation langfristig als Durch den Prozess der nutzerzentrierten Softwareentwicklung sammeln Organisationen wichtige Erkenntnisse über die Nutzer ihrer Produkte, deren Arbeitsaufgaben und über die Nutzungskontexte, in denen sie angewendet werden. Diese Arbeit untersucht, wie derartige Usability-Ergebnisse in einer Organisation langfristig als Wissensressource eingesetzt werden können, um die Usability zukünftiger Produkte zu verbessern und die Effizienz des nutzerzentrierten Entwicklungsprozesses zu optimieren. Im Fokus stehen dabei interne Usability-Beauftragte als Anwender dieser Wissensressource: Da diese innerhalb ihrer Organisation für die dort entwickelten Produkte verantwortlich sind, haben sie ein besonders hohes Interesse an der nachhaltigen Nutzung der erhobenen Usability-Ergebnisse. Zu einer organisationsinternen Nutzung von Usability-Ergebnissen existieren bereits Ansätze aus der Forschung zu nutzerzentrierten Entwicklungsprozessen im Bereich der Mensch-Computer-Interaktion, die unterschiedliche Ziele verfolgen. (Hughes, 2006; Douglas, 2007; Vilbergsdottir et al., 2014). Einen frühen Ansatz stellt Andre et al. (2001) mit dem User Action Framework vor, dessen Anwendung jedoch aufwendig sein kann (Hornbæk et al., 2008). Vorschläge für die Klassifizierung von empirischen Usability-Ergebnissen werden bislang vor allem im Kontext der Forschung zu Usability-Methoden eingesetzt (etwa Lavery et al., 1997; Hornbæk et al., 2008). In anderen Bereichen finden sie jedoch keine breite Anwendung, da sie nicht auf die Anwendungsfälle für Usability-Information in Organisationen abgestimmt sind. Als eine zentrale Fragestellung dieser Arbeit wird daher untersucht, für welche Anwendungsfälle eine Sammlung von Usability-Ergebnissen eingesetzt werden kann (Forschungsfrage RQ1). Dafür werden qualitativ ausgerichtete Interviews (n=8) mit internen Usability-Beauftragten sowie Fokusgruppen in zwei Organisationen durchgeführt. Im Rahmen dieser Studien können außerdem die Anforderungen an die Wissensorganisation und an die Informationsinteraktion für die Nutzung von Usability-Ergebnissen als Wissensressource analysiert werden (RQ2). Die Anforderungen werden als ein prototypisches Usability-Informationssystem umgesetzt, welches den Zugang zu einer Sammlung von Usability-Ergebnissen bereitstellt. In einer Studie mit Usability-Beauftragten (n=11) wird dieses System evaluiert, um Rückschlüsse auf die zugrunde liegenden Anforderungen zu ermöglichen. Im Rahmen der Studie werden zudem die Entscheidungsprozesse diskutiert, die angewendet werden, wenn Usability-Ergebnisse auf andere Kontexte übertragen oder verallgemeinert werden sollen (RQ3). Weiterhin werden die Faktoren und Barrieren untersucht, welche die Akzeptanz von Usability-Ergebnissen als Wissensressource in einer Organisation beeinflussen (RQ4). Die Untersuchungen zeigen, dass Usability-Ergebnisse bereits in vielen Organisationen gesammelt und gezielt eingesetzt werden. Die erhobenen Anwendungsfälle (RQ1) umfassen die Übertragung von vorhandenen Ergebnissen auf aktuelle Gestaltungsentscheidungen, Lernprozesse, analytische Fragestellungen und die Verallgemeinerung zu internen Richtlinien. Zu den identifizierten Anforderungen für die Organisation von Usability-Wissen (RQ2) gehört die Kombination von produktübergreifenden und produktbezogenen Metadaten. Die empirischen Evaluierungsergebnisse aus Nutzertests sollten mit den zugrunde liegenden Daten, vor allem aber mit den resultierenden Lösungsvorschlägen verknüpft werden. Bei der Gestaltung der Informationsinteraktion sollten die gezielte Suche, der Umgang mit potenziell unbekannter oder wechselnder Terminologie, aber auch explorative Such- und Lernprozesse unterstützen werden. Wenn Usability-Ergebnisse in einer Organisation mit dem Ziel der Vollständigkeit erhoben werden, können darauf auch Funktionen für die quantitative Analyse und für die Prozessbewertung aufbauen. Für die Bewertung der Übertragbarkeit von Usability-Ergebnissen (RQ3) sind eine Reihe von Entscheidungskriterien und Hinweisen relevant, anhand derer ihre Zuverlässigkeit überprüft und der Erhebungskontext hinsichtlich der Relevanz für eine aktuelle Fragestellung bewertet werden kann. Die Akzeptanz der Anwendung von Usability-Wissen (RQ4) erscheint primär von dem Aufwand abhängig, der für die Erschließung der Ergebnisse erforderlich ist. Die meisten der Teilnehmer bewerten den Aufwand im Verhältnis zu den erwarteten Vorteilen jedoch als angemessen. Mögliche Barrieren für die Wissensteilung können aus der Befürchtung entstehen, die Kontrolle über die Interpretation der Ergebnisse zu verlieren, sowie aus der Wahrnehmung als öffentliche Kritik an den jeweiligen Produktverantwortlichen. Die Ergebnisse dieser Arbeit können dabei helfen, die Unterstützung für die Nutzung von Usability-Ergebnissen als Wissensressource auf die erhobenen Anwendungsfälle auszurichten. Dafür werden Empfehlungen zu möglichen Ausrichtungen eines Usability-Informationssystems in Organisationen gegeben. Die Ergebnisse verweisen außerdem auf das große Potenzial für weitere Forschungsvorhaben in diesem Bereich, sowohl in Hinblick auf eine bessere Unterstützung des Wissensmanagements von Usability-Ergebnissen als auch in Bezug auf die Übertragung der grundlegenden Erkenntnisse dieser Arbeit auf andere Anwendungsdomänen, etwa im Bereich des Managements von Forschungsdaten.User centered software development provides organizations with valuable insights about the users of their software, about their work tasks and the various contexts in which a product is used. This dissertation explores how organizations can profit even more from such results in the long term by using them as an internal knowledge resource for improving the usability of future products and for increasing the efficiency of user centered processes. This topic will be investigated for in-house usability consultants as the primary target group of such a resource. In-house consultants are responsible for the quality of the products developed in their company, and the sustainable management of internal usability results therefore is of particular interest to them. In the research field of human computer interaction and user centered design, several approaches have already been proposed which can be used to systematize usability results in order to pursue a variety of goals (e.g. Hughes 2006; Douglas 2007; Vilbergsdottir et al. 2014). The User Action Framework (Andre et al. 2001) is an important contribution in this area. Its implementation, however, may prove to be difficult for many organizations because applying it was found to be resource intensive (Hornbæk et al., 2008). Other classification systems for usability problems have predominantly been in use in scientific studies on the evaluation of usability methods (e. g. Lavery et al. 1997; Hornbæk & Frøkjær 2008). These approaches have not been widely adopted because of the efforts involved in applying them, and because they do not take into account relevant use cases for usability information in organizations. The identification of use cases for the internal application of usability results therefore constitutes an important research question of this dissertation (research question RQ1). Qualitative interviews with in-house usability consultants (n=8) as well as focus groups in two organizations are conducted in order to investigate this question and to elicit usage requirements of an usability information system (research question RQ2). A prototypical usability information system implements these requirements based on a set of realistic usability results. The system and the proposed requirements are evaluated in an additional study with usability consultants (n=11). In the context of this study, criteria for reusing and generalizing usability results can be examined from the point of view of the participants (research question RQ3). In addition, the factors and barriers influencing the process of sharing and using usability knowledge have been investigated (research question RQ4). Results demonstrate that usability results have already been collected and applied to different use cases in many organizations (RQ1), including their direct application to current design decisions, learning and exploration, analytic questions, and the creation of internal usability standards. The organization of usability results (RQ2) requires a combination of product-specific characteristics with more general attributes as metadata for search and analysis. Results from user studies should be linked to the underlying empirical data and to the resulting design recommendations. Requirements for information interaction include support for the targeted search for usability results, dealing with potentially unknown or changing terminology, as well as possibilities for exploratory search and learning. If results are collected comprehensively in an organization, features for information analysis can be used to support the improvement of development processes. A number of different criteria are used to assess the reliability of usability results and the fit between the context in which a result was elicited and the context to which the result is to be applied. These aspects together provide the basis for deciding about the transferability of results (RQ3). Acceptance of the application of usability results as an information resource (RQ4) primarily depends on the amount of effort which is required for documenting these results. However, most participants expect the benefits to outweigh these efforts. Possible barriers for sharing usability results also include concerns about the loss of control over their interpretation as well as the perception of published results as criticism by those who are responsible for a product. In addition to describing existing practices, the results of this dissertation are intended to offer assistance for the application of usability results as an information resource in different use cases. Accordingly, recommendations about different categories of usability information systems are presented. The findings indicate further possibilities for research with the goal of improving knowledge management for usability results and may also be applied to other domains such as research data management

Between &#039;technological obduracy&#039; and &#039;academic resistance&#039;: concepts of use of blackboard and the experience of university teachers.

Until recently, Blackboard has been one of the most common forms of learning-management systems (LMSs) in use in Australian universities. However, it has been adopted and used by academics far less than its proponents had anticipated. The literature of academic use of learning-management systems paints a picture, either of a relatively straightforward understanding of adoption of new educational technologies as an informational problem, or alternatively, of problematic academics who ‘resist’ using Blackboard. Academics themselves can understand the technology of Blackboard to be obdurate, time consuming and difficult to use. Drawing on a combination of sensemaking theory, practice theory and the socio-technical theories of social construction of technology and actor network theory, I ask how academics have set about using Blackboard. I clarify how educational technology use in the literature is constructed from diverse perspectives and how users in practice negotiate their way through Blackboard at four levels of encounter: as material infrastructure; as a process of orientation to, and reading of, navigational and interface symbolism; as an individual sensemaking project about representing education; and as an organisational representation and a technical system. Each level contains capacities for disruption and rebuilding of former habits and sense. “Rebuilding” a new interpretation and an effective use of Blackboard by any individual academic is never certain, as at each level different strategies are required, but rebuilding a notion of technologised education by creating ”genres of use” explains some of the differential in approaches to Blackboard use. At a meta level, this process of creating “use” also elucidates theories of action, practice and social change in practice theory and to a lesser extent in social construction of technology and actor network theory, by adding the insights of sensemaking theory to show how academics build their own concepts of use in an LMS, that can appear obdurate and unwieldy to users. The theoretical purpose is to offer an essay in understanding the processes of socio-technical change where change is not necessarily fostered by technological ease or user self-motivation