On the Use of Experiments in Design Science Research: A Proposition of an Evaluation Framework

Although experiments are considered a valid scientific method for evaluating the outcome of design science research (DSR), only minimal procedural guidelines or standards exist that help researchers in the setup and conduct. To take advantage of and encourage researchers to include design experiments as an inherent part of their build and test cycle, this study proposes a set of guidelines. In order to get a broad overview of how researchers currently apply the experimental method in DSR and to detect potential drawbacks, an extensive review of the extant literature was conducted. On this basis, we propose an evaluation framework that complements the general design science research guidelines of Hevner and colleagues. The purpose of this framework is to assist researchers, reviewers, editors, and readers in understanding possible pitfalls as well as to ask the right questions which need to be answered in the conduct of design experiments

Evaluating epistemic uncertainty under incomplete assessments

The thesis of this study is to propose an extended methodology for laboratory based Information Retrieval evaluation under incomplete relevance assessments. This new methodology aims to identify potential uncertainty during system comparison that may result from incompleteness. The adoption of this methodology is advantageous, because the detection of epistemic uncertainty - the amount of knowledge (or ignorance) we have about the estimate of a system's performance - during the evaluation process can guide and direct researchers when evaluating new systems over existing and future test collections. Across a series of experiments we demonstrate how this methodology can lead towards a finer grained analysis of systems. In particular, we show through experimentation how the current practice in Information Retrieval evaluation of using a measurement depth larger than the pooling depth increases uncertainty during system comparison

COalitions in COOperation Networks (COCOON):Social Network Analysis and Game Theory to Enhance Cooperation Networks

Sie, R. L. L. (2012). COalitions in COOperation Networks (COCOON): Social Network Analysis and Game Theory to Enhance Cooperation Networks (Unpublished doctoral dissertation). September, 28, 2012, Open Universiteit in the Netherlands (CELSTEC), Heerlen, The Netherlands.IdSpace, SIK

On Supporting Android Software Developers And Testers

Users entrust mobile applications (apps) to help them with different tasks in their daily lives. However, for each app that helps to finish a given task, there are a plethora of other apps in popular marketplaces that offer similar or nearly identical functionality. This makes for a competitive market where users will tend to favor the highest quality apps in most cases. Given that users can easily get frustrated by apps which repeatedly exhibit bugs, failures, and crashes, it is imperative that developers promptly fix problems both before and after the release. However, implementing and maintaining high quality apps is difficult due to unique problems and constraints associated with the mobile development process such as fragmentation, quick feature changes, and agile software development. This dissertation presents an empirical study, as well as several approaches for developers, testers and designers to overcome some of these challenges during the software development life cycle. More specifically, first we perform an in-depth analysis of developers’ needs on automated testing techniques. This included surveying 102 contributors of open source Android projects about practices for testing their apps. The major findings from this survey illustrate that developers: (i) rely on usage models for designing test app cases, (ii) prefer expressive automated generated test cases organized around use cases, (iii) prefer manual testing over automation due to reproducibility issues, and (iv) do not perceive that code coverage is an important measure of test case quality. Based on the findings from the survey, this dissertation presents several approaches to support developers and testers of Android apps in their daily tasks. In particular, we present the first taxonomy of faults in Android apps. This taxonomy is derived from a manual analysis of 2,023 software artifacts extracted from six different sources (e.g., stackoverflow and bug reports). The taxonomy is divided into 14 categories containing 262 specific types. Then, we derived 38 Android-specific mutation operators from the taxonomy. Additionally, we implemented the infrastructure called MDroid+ that automatically introduces mutations in Android apps. Third, we present a practical automation for crowdsourced videos of mobile apps called V2S. This solution automatically translates video recordings of mobile executions into replayable user scenarios. V2S uses computer vision and adopts deep learning techniques to identify user interactions from video recordings that illustrate bugs or faulty behaviors in mobile apps. Last but not least, we present an approach that aims at supporting the maintenance process by facilitating the way users report bugs for Android apps. It comprises the interaction between an Android and a web app that assist the reporter by automatically collecting relevant information

Innovation of product modularity development through the integration of a formal Industrial Design framework

Growing numbers of global manufacturers are not only adopting the modularity concept, but integrating design methodologies that explicitly focused on achieving a range of competitive advantages through the enhancement of product appearance and utilities designs. The rising interest in industrial design is also an interesting symptom of changes in the approach to new product development, hence, integrating industrial design in modular product design posed a new challenge. In meeting these challenges, a formal Industrial Design framework known as InDFM (Industrial Design Framework for Modular Product Design/Development) was developed to support the innovation of design in modular product development. Within the InDFM, a methodology is developed for modular product design realisation. This research embarked with identifying the appropriate range of product as the focus of the investigation, followed by qualitative surveys on the design and development processes relevant to the selected product. The surveys were conducted in modular product companies within a range of industries related to the product, in the U.K., Belgium and Malaysia. Literatures reviews were also conducted on related domains across a range of application to understand the fundamentals of modularity and industrial design processes that are relevant to the domains. Data findings from these exercises were used to identify InDFM construction components, which were also vital to develop a technical standard for implementation of the InDFM. To evaluate its practicability, the InDFM was retrospectively applied in an existing modular product design process of a selected company. The evaluation focused on process compatibility of industrial design and modular design processes. Validation of the process compatibility emphasised the quality of integration at all stages of the design and development process. In conclusion, industrial design applications in a highly technical process of modular product design provide a design-driven innovation to complement the engineering driven innovation in the process. The combinations were proven to enhance the visual, interactive, and the feasibility contents of a modular product apart from providing a broader perspective to the objective of product modularity. InDFM also provides design practitioners with systematic design methodology to integrate both processes, thus performed as a tool for innovation that facilitate the revision of object identity, break away from the existing design rules and generating new rules. Additionally, as InDFM is a flexible methodology, innovation of modular product design through industrial design is accessible to any product company, small scale or big organisation that would want to acquire an advanced interactive version of the InDFM in the future

Probabilistic Human-Robot Information Fusion

This thesis is concerned with combining the perceptual abilities of mobile robots and human operators to execute tasks cooperatively. It is generally agreed that a synergy of human and robotic skills offers an opportunity to enhance the capabilities of today’s robotic systems, while also increasing their robustness and reliability. Systems which incorporate both human and robotic information sources have the potential to build complex world models, essential for both automated and human decision making. In this work, humans and robots are regarded as equal team members who interact and communicate on a peer-to-peer basis. Human-robot communication is addressed using probabilistic representations common in robotics. While communication can in general be bidirectional, this work focuses primarily on human-to-robot information flow. More specifically, the approach advocated in this thesis is to let robots fuse their sensor observations with observations obtained from human operators. While robotic perception is well-suited for lower level world descriptions such as geometric properties, humans are able to contribute perceptual information on higher abstraction levels. Human input is translated into the machine representation via Human Sensor Models. A common mathematical framework for humans and robots reinforces the notion of true peer-to-peer interaction. Human-robot information fusion is demonstrated in two application domains: (1) scalable information gathering, and (2) cooperative decision making. Scalable information gathering is experimentally demonstrated on a system comprised of a ground vehicle, an unmanned air vehicle, and two human operators in a natural environment. Information from humans and robots was fused in a fully decentralised manner to build a shared environment representation on multiple abstraction levels. Results are presented in the form of information exchange patterns, qualitatively demonstrating the benefits of human-robot information fusion. The second application domain adds decision making to the human-robot task. Rational decisions are made based on the robots’ current beliefs which are generated by fusing human and robotic observations. Since humans are considered a valuable resource in this context, operators are only queried for input when the expected benefit of an observation exceeds the cost of obtaining it. The system can be seen as adjusting its autonomy at run-time based on the uncertainty in the robots’ beliefs. A navigation task is used to demonstrate the adjustable autonomy system experimentally. Results from two experiments are reported: a quantitative evaluation of human-robot team effectiveness, and a user study to compare the system to classical teleoperation. Results show the superiority of the system with respect to performance, operator workload, and usability

Mitigating security and privacy threats from untrusted application components on Android

Aufgrund von Androids datenzentrierter und Open-Source Natur sowie von fehlerhaften/bösartigen Apps durch das lockere Marktzulassungsverfahren, ist die Privatsphäre von Benutzern besonders gefährdet. Diese Dissertation präsentiert eine Reihe von Forschungsarbeiten, die die Bedrohung der Sicherheit/Privatsphäre durch nicht vertrauenswürdige Appkomponenten mindern. Die erste Arbeit stellt eine Compiler-basierte Kompartmentalisierungslösung vor, die Privilegientrennung nutzt, um eine starke Barriere zwischen der Host-App und Bibliothekskomponenten zu etablieren, und somit sensible Daten vor der Kompromittierung durch neugierige/bösartige Werbe-Bibliotheken schützt. Für fehleranfällige Bibliotheken von Drittanbietern implementieren wir in der zweiten Arbeit ein auf API-Kompatibilität basierendes Bibliothek-Update-Framework, das veraltete Bibliotheken durch Drop-Ins aktualisiert, um das durch Bibliotheken verursachte Zeitfenster der Verwundbarkeit zu minimieren. Die neueste Arbeit untersucht die missbräuchliche Nutzung von privilegierten Accessibility(a11y)-Funktionen in bösartigen Apps. Wir zeigen ein datenschutzfreundliches a11y-Framework, das die a11y-Logik wie eine Pipeline behandelt, die aus mehreren Modulen besteht, die in verschiedenen Sandboxen laufen. Weiterhin erzwingen wir eine Flusskontrolle über die Kommunikation zwischen den Modulen, wodurch die Angriffsfläche für den Missbrauch von a11y-APIs verringert wird, während die Vorteile von a11y erhalten bleiben.While Android’s data-intensive and open-source nature, combined with its less-than-strict market approval process, has allowed the installation of flawed and even malicious apps, its coarse-grained security model and update bottleneck in the app ecosystem make the platform’s privacy and security situation more worrying. This dissertation introduces a line of works that mitigate privacy and security threats from untrusted app components. The first work presents a compiler-based library compartmentalization solution that utilizes privilege separation to establish a strong trustworthy boundary between the host app and untrusted lib components, thus protecting sensitive user data from being compromised by curious or malicious ad libraries. While for vulnerable third-party libraries, we then build the second work that implements an API-compatibility-based library update framework using drop-in replacements of outdated libraries to minimize the open vulnerability window caused by libraries and we perform multiple dynamic tests and case studies to investigate its feasibility. Our latest work focuses on the misusing of powerful accessibility (a11y) features in untrusted apps. We present a privacy-enhanced a11y framework that treats the a11y logic as a pipeline composed of multiple modules running in different sandboxes. We further enforce flow control over the communication between modules, thus reducing the attack surface from abusing a11y APIs while preserving the a11y benefits

Usability inspection for sonification applications

Bonification is the representation of data using mainly non-speech sound for the purpose of communication and interpretation. The process and technique of converting the data into sound is called the sonification technique. One or more techniques might be required by a sonification application. However, sonification techniques are not generally suitable for all kinds of data, and often custom techniques are used - where the design is tailored to the domain and nature of the data as well as the users' required tasks within the application. Therefore, it is important to assure the usability of the technique for the specific domain application being developed. In previously reported research, most designers of sonification applications have needed to develop at least a prototype for user testing. The result are interpreted and analysed to look for potential problems and solutions to improve the design. This dissertation has developed a new systematic usability inspection approach called the Task Interpretation Walkthrough (TIW) for the design of sonification application before they go to the initial development phase. It is hypothesized that designers of sonification applications will be able to detect significantly more important potential usability problems before the implementation phase by analysing the interaction between the user and the application as well as paying attention to the different stages of how the data is transformed into sound. It uses two new models - the Sonification Application (SA) model and the User Interpretation Construction (UIC) model. Four experiments with human subjects were carried out to study the feasibility and effectiveness of Task Interpretation Walkthrough inspection by comparing it against two widely used techniques; Heuristic Evaluation and Cognitive Walkthrough. The sonification designs being inspected were a Mobile Phone Joystick Text-Entry with Sound (Experiments I and II), a Diagnosis Tool for Analysis of The Motion and Usage of a Patient's Arm (Experiment III); and an Audio-Visual Analysis Tool of Cervical Sample Slides (Experiment IV). The participants included sound researchers (Experiment II); and students with a background in music technology and software engineering (Experiments I, III and IV), acting either individually or in 2-person groups. The results have shown that the research hypothesis is supported, where the significantly important usability problems were able to be detected before the implementation phase. From the inspection method comparison study, results showed the Task Interpretation Walkthrough to be more effective than the existing techniques (Heuristic Evaluation and Cognitive Walkthrough)

Towards improved decision support in the assessment and management of pain for people with dementia in hospital: a systematic meta-review and observational study

This is the final version. Available from NIHR Journals Library via the DOI in this record.Background Pain and dementia are common in older people, and impaired cognitive abilities make it difficult for them to communicate their pain. Pain, if poorly managed, impairs health and well-being. Accurate pain assessment in this vulnerable group is challenging for hospital staff, but essential for appropriate management. Robust methods for identifying, assessing and managing pain are needed. Aims and objectives Two studies were undertaken to inform the development of a decision support tool to aid hospital staff in the recognition, assessment and management of pain. The first was a meta-review of systematic reviews of observational pain assessment instruments with three objectives: (1) to identify the tools available to assess pain in adults with dementia; (2) to identify in which settings they were used and with what patient populations; and (3) to assess their reliability, validity and clinical utility. The second was a multisite observational study in hospitals with four objectives: (1) to identify information currently used by clinicians when detecting and managing pain in patients with dementia; (2) to explore existing processes for detecting and managing pain in these patients; (3) to identify the role (actual/potential) of carers in this process; and (4) to explore the organisational context in which health professionals operate. Findings also informed development of health economics data collection forms to evaluate the implementation of a new decision support intervention in hospitals. Methods For the meta-review of systematic reviews, 12 databases were searched. Reviews of observational pain assessment instruments that provided psychometric data were included. Papers were quality assessed and data combined using narrative synthesis. The observational study used an ethnographic approach in 11 wards in four UK hospitals. This included non-participant observation of 31 patients, audits of patient records, semistructured interviews with 52 staff and four carers, informal conversations with staff and carers and analysis of ward documents and policies. Thematic analysis of the data was undertaken by the project team. Results Data from eight systematic reviews including 28 tools were included in the meta-review. Most tools showed moderate to good reliability, but information about validity, feasibility and clinical utility was scarce. The observational study showed complex ward cultures and routines, with variations in time spent with patients, communication patterns and management practices. Carer involvement was rare. No pain decision support tools were observed in practice. Information about pain was elicited in different ways, at different times, by different health-care staff and recorded in separate documents. Individual staff made sense of patients’ pain by creating their own ‘overall picture’ from available information. Limitations Grey literature and non-English-language papers were excluded from the meta-review. Sample sizes in the observational study were smaller than planned owing to poor documentation of patients’ dementia diagnoses, gatekeeping by staff and difficulties in gaining consent/assent. Many patients had no or geographically distant carers, or a spouse who was too unwell and/or reluctant to participate. Conclusions No single observational pain scale was clearly superior to any other. The traditional linear concept of pain being assessed, treated and reassessed by single individuals did not ‘fit’ with clinical reality. A new approach enabling effective communication among patients, carers and staff, centralised recording of pain-related information, and an extended range of pain management interventions is proposed [Pain And Dementia Decision Support (PADDS)]. This was not tested with users, but a follow-on study aims to codesign PADDS with carers and clinicians, then introduce education on staff/patient/carer communications and use of PADDS within a structured implementation plan. PADDS will need to be tested in differing ward contexts.National Institute for Health Research Health Services and Delivery Research programm