Controlled Experimentation in Naturalistic Mobile Settings

Performing controlled user experiments on small devices in naturalistic mobile settings has always proved to be a difficult undertaking for many Human Factors researchers. Difficulties exist, not least, because mimicking natural small device usage suffers from a lack of unobtrusive data to guide experimental design, and then validate that the experiment is proceeding naturally.Here we use observational data to derive a set of protocols and a simple checklist of validations which can be built into the design of any controlled experiment focused on the user interface of a small device. These, have been used within a series of experimental designs to measure the utility and application of experimental software. The key-point is the validation checks -- based on the observed behaviour of 400 mobile users -- to ratify that a controlled experiment is being perceived as natural by the user. While the design of the experimental route which the user follows is a major factor in the experimental setup, without check validations based on unobtrusive observed data there can be no certainty that an experiment designed to be natural is actually progressing as the design implies.Comment: 12 pages, 3 table

To deceive or not to deceive! Legal implications of phishing covert research

Whilst studying mobile users' susceptibility to phishing attacks, we found ourselves subject to regulations concerning the use of deception in research. We argue that such regulations are misapplied in a way that hinders the progress of security research. Our argument analyses the existing framework and the ethical principles of conducting phishing research in light of these regulations. Building on this analysis and reflecting on real world experience; we present our view of good practice and suggest guidance on how to prepare legally compliant proposals to concerned ethics committee

Teamwork Evaluation with a Microworld Platform

Trabalho apresentado em IEEE 20th International Conference on Computer Supported Cooperative Work in Design,4-6 maio 2016, Nanchang, ChinaN/

“Natural Laboratory Complex” for novel primate neuroscience

We propose novel strategies for primate experimentation that are ethically valuable and pragmatically useful for cognitive neuroscience and neuropsychiatric research. Specifically, we propose Natural Laboratory Complex or Natural Labs, which are a combination of indoor-outdoor structures for studying free moving and socially housed primates in natural or naturalistic environment. We contend that Natural Labs are pivotal to improve primate welfare, and at the same time to implement longitudinal and socio-ecological studies of primate brain and behavior. Currently emerging advanced technologies and social systems (including recent COVID-19 induced “remote” infrastructures) can speed-up cognitive neuroscience approaches in freely behaving animals. Experimental approaches in natural(istic) settings are not in competition with conventional approaches of laboratory investigations, and could establish several benefits at the ethical, experimental, and economic levels.Introduction Animal models in neuroscience - The rodent model - The non-human primate model - Optimizing cognitive neuroscience research with animal models Novel strategies for primate experimentation - Natural laboratory complex -- In situ Lab-in-Nature -- Ex situ Nature-in-Lab Harmonization of cost and benefit trade-offs - Ethical balance - Socioeconomic balance - Legal balance Conclusio

Organizational Ethnography and the Art of Judgment in-the-Moment

This paper responds to the conveners’ call on exploring and advancing Organizational Ethnography (OE) as a paradigm for the organizational sciences. This sub-theme is linked through my empirical study of senior managers in international development organizations and how they make sense of using their judgment ‘in-the-moment’ in the context of their leadership roles and work environments. I adopt an “ethnographic orientation” (Watson, 2011, p.216) and emphasize the need for a highly reflexive approach in an ethnographer’s role as making judgments throughout the challenging processes of doing “fieldwork, headwork and textwork” (Van Maanen, 2011, p.218). Theoretically, this study contributes to the existing judgment and decision making literature from a social constructionist perspective by drawing linkages to judgment as a co-constructed phenomenon. How senior managers understand their judgment-making in situations ‘in-the-moment’ is an understudied area thus far and even scarcer in the context of international development organizations. Methodologically, the ethnographic and radically reflexive approach taken addresses a gap in the literature, builds awareness and raises in importance examining the ‘multiple selves’ (Reinharz, 1997) of the ethnographer. How I influenced my research and was influenced by it as both subject and object were key to my findings. In addition to the sub-theme call, this paper also links to the overarching Colloquium theme, ‘Bridging Continents, Cultures and Worldviews’, by connecting the cultures and co- constructed views of the researcher and practitioners. The collaborative, ethnographic approach taken was a unique way to get ‘up close and personal’ in understanding what judgment meant to senior leaders in the two participating UK-headquartered organizations. With international development missions in African nations, the senior leaders continuously constructed their own bridges across borders in their financing, operations and communications between their team members and external stakeholders located in multiple countries, reliant on virtual offices and mobile and Internet technology to stay connected. My judgment as an ethnographer was necessary to determine how to best embrace this way of ‘working’ during fieldwork and become another type of stakeholder to them. I will begin with a brief theoretical and methodological background of my exploratory study, identifying the gap in the literature and how my study fills it. Then I will outline the methodology, methods, data collection and analysis and findings. Finally, I will conclude with the challenges of ‘doing organizational ethnography’ inside small international development organizations and the contributions made to advance OE as a unique way to study the social phenomenon of judgment ‘in-the moment’

Beyond the RCT: Integrating Rigor and Relevance to Evaluate the Outcomes of Domestic Violence Programs

Programs for domestic violence (DV) victims and their families have grown exponentially over the last four decades. The evidence demonstrating the extent of their effectiveness, however, often has been criticized as stemming from studies lacking scientific rigor. A core reason for this critique is the widespread belief that credible evidence can derive only from research grounded in randomized control trials (RCTs). Although the RCT method has its strengths, we argue that it is rarely an optimal—or even a possible—approach for evaluating multifaceted DV programs. This article reviews the reasons that RCT is a poor fit for such programs and argues that a more inclusive conceptualization of credible evidence is critical to expanding our knowledge base about how DV programs affect survivors’ safety and well-being

A Review of Research Methodologies Employed in Serendipity Studies in the Context of Information Research

Background: The concept of serendipity has become increasingly interesting for those undertaking serendipity research in recent years. However, serendipitous encounters are subjective and rare in a real-world context, making this an extremely challenging subject to study. Methods: Various methods have been proposed to enable researchers to understand and measure serendipity, but there is no broad consensus on which methods to use in different experimental settings. A comprehensive literature review was first conducted, which summarizes the research methods being employed to study serendipity. It was followed by a series of interviews with experts that specified the relative strengths and weaknesses of each method identified in the literature review, in addition to the challenges usually confronted in serendipity research. Results: The findings suggest using mixed research methods to produce a more complete picture of serendipity and contribute to the verification of any research findings. Several challenges and implications relating to empirical studies in the investigation of serendipity have been derived from this study. Conclusions: This paper investigated research methods employed to study serendipity by synthesizing finding from a literature review and the interviews with experts. It provides a methodological contribution to serendipity studies by systematically summarizing the methods employed in the studies of serendipity and identifying the strengths and weakness of each method. It also suggests the novel approach of using mixed research methods to study serendipity. This study has potential limitations related to a small number of experts involved in the expert interview. However, it should be noted that the nature of the topic is a relatively focused area, and it was observed after interviewing the experts that new data seems to not contribute to the findings owing to its repetition of comment

Experiential design landscapes: design research in the wild

Thanks to the emergence of new sensing and behaviour tracking technologies, design research can take place anywhere and anytime in the real world. When doing design research, a trade-off has to be made between experimental control and ecological validity. In this paper, we compare Experiential Design Landscapes (EDLs) with three more traditional research approaches that are frequently used in design research, i.e., Lab Research, Living Lab and design research ‘in the field’, and reflect on this trade-off. By means of an example, we discuss how EDLs deals with issues of ‘generalisability’ to the real world and the potential loss of experimental control

Modified Structured Domain Randomization in a Synthetic Environment for Learning Algorithms

Deep Reinforcement Learning (DRL) has the capability to solve many complex tasks in robotics, self-driving cars, smart grids, finance, healthcare, and intelligent autonomous systems. During training, DRL agents interact freely with the environment to arrive at an inference model. Under real-world conditions this training creates difficulties of safety, cost, and time considerations. Training in synthetic environments helps overcome these difficulties, however, this only approximates real-world conditions resulting in a ‘reality gap’. The synthetic training of agents has proven advantageous but requires methods to bridge this reality gap. This work addressed this through a methodology which supports agent learning. A framework which incorporates a modifiable synthetic environment integrated with an unmodified DRL algorithm was used to train, test, and evaluate agents while using a modified Structured Domain Randomization (SDR+) technique. It was hypothesized that the application of environment domain randomizations (DR) during the learning process would allow the agent to learn variability and adapt accordingly. Experiments using the SDR+ technique included naturalistic and physical-based DR while applying the concept of context-aware elements (CAE) to guide and speed up agent training. Drone racing served as the use case. The experimental framework workflow generated the following results. First, a baseline was established by training and validating an agent in a generic synthetic environment void of DR and CAE. The agent was then tested in environments with DR which showed degradation of performance. This validated the reality gap phenomenon under synthetic conditions and established a metric for comparison. Second, an SDR+ agent was successfully trained and validated under various applications of DR and CAE. Ablation studies determine most DR and CAE effects applied had equivalent effects on agent performance. Under comparison, the SDR+ agent’s performance exceeded that of the baseline agent in every test where single or combined DR effects were applied. These tests indicated that the SDR+ agent’s performance did improve in environments with applied DR of the same order as received during training. The last result came from testing the SDR+ agent’s inference model in a completely new synthetic environment with more extreme and additional DR effects applied. The SDR+ agent’s performance was degraded to a point where it was inconclusive if generalization occurred in the form of learning to adapt to variations. If the agent’s navigational capabilities, control/feedback from the DRL algorithm, and the use of visual sensing were improved, it is assumed that future work could exhibit indications of generalization using the SDR+ technique

Citizen Involvement in Service Co-creation in Urban Living Labs

Urban Living Lab (ULL) is a living lab in which citizens and companies collaborate to create services for solving problems in a city or region. In ULLs, a variety of citizens participate in a long-term co-creation process including design activities such as concept creation, development, and testing. Unfortunately, few studies have provided useful knowledge about or insights into how to effectively involve citizens with diverse characteristics in such co-creation processes. In this paper, we present a case study illustrating how to involve various citizens in the long-term co-creative design process in ULLs. In this study, we first analyze our ULL project and clarify the various roles that citizens may perform in the co-creation process. Then, on the basis of the analysis results as well as our hands-on experiences, we provide key insights into obtaining effective citizen involvement in ULLs, which should be helpful to other practitioners and researchers