Criteria for Rigor in Visualization Design Study

We develop a new perspective on research conducted through visualization design study that emphasizes design as a method of inquiry and the broad range of knowledge-contributions achieved through it as multiple, subjective, and socially constructed. From this interpretivist position we explore the nature of visualization design study and develop six criteria for rigor. We propose that rigor is established and judged according to the extent to which visualization design study research and its reporting are INFORMED, REFLEXIVE, ABUNDANT, PLAUSIBLE, RESONANT, and TRANSPARENT. This perspective and the criteria were constructed through a four-year engagement with the discourse around rigor and the nature of knowledge in social science, information systems, and design. We suggest methods from cognate disciplines that can support visualization researchers in meeting these criteria during the planning, execution, and reporting of design study. Through a series of deliberately provocative questions, we explore implications of this new perspective for design study research in visualization, concluding that as a discipline, visualization is not yet well positioned to embrace, nurture, and fully benefit from a rigorous, interpretivist approach to design study. The perspective and criteria we present are intended to stimulate dialogue and debate around the nature of visualization design study and the broader underpinnings of the discipline

Improving transparency and scientific rigor in academic publishing.

Progress in basic and clinical research is slowed when researchers fail to provide a complete and accurate report of how a study was designed, executed, and the results analyzed. Publishing rigorous scientific research involves a full description of the methods, materials, procedures, and outcomes. Investigators may fail to provide a complete description of how their study was designed and executed because they may not know how to accurately report the information or the mechanisms are not in place to facilitate transparent reporting. Here, we provide an overview of how authors can write manuscripts in a transparent and thorough manner. We introduce a set of reporting criteria that can be used for publishing, including recommendations on reporting the experimental design and statistical approaches. We also discuss how to accurately visualize the results and provide recommendations for peer reviewers to enhance rigor and transparency. Incorporating transparency practices into research manuscripts will significantly improve the reproducibility of the results by independent laboratories

How software engineering research aligns with design science: A review

Background: Assessing and communicating software engineering research can be challenging. Design science is recognized as an appropriate research paradigm for applied research but is seldom referred to in software engineering. Applying the design science lens to software engineering research may improve the assessment and communication of research contributions. Aim: The aim of this study is 1) to understand whether the design science lens helps summarize and assess software engineering research contributions, and 2) to characterize different types of design science contributions in the software engineering literature. Method: In previous research, we developed a visual abstract template, summarizing the core constructs of the design science paradigm. In this study, we use this template in a review of a set of 38 top software engineering publications to extract and analyze their design science contributions. Results: We identified five clusters of papers, classifying them according to their alignment with the design science paradigm. Conclusions: The design science lens helps emphasize the theoretical contribution of research output---in terms of technological rules---and reflect on the practical relevance, novelty, and rigor of the rules proposed by the research.Comment: 32 pages, 10 figure

Towards Convergence: How to Do Transdisciplinary Environmental Health Disparities Research.

Increasingly, funders (i.e., national, public funders, such as the National Institutes of Health and National Science Foundation in the U.S.) and scholars agree that single disciplines are ill equipped to study the pressing social, health, and environmental problems we face alone, particularly environmental exposures, increasing health disparities, and climate change. To better understand these pressing social problems, funders and scholars have advocated for transdisciplinary approaches in order to harness the analytical power of diverse and multiple disciplines to tackle these problems and improve our understanding. However, few studies look into how to conduct such research. To this end, this article provides a review of transdisciplinary science, particularly as it relates to environmental research and public health. To further the field, this article provides in-depth information on how to conduct transdisciplinary research. Using the case of a transdisciplinary, community-based, participatory action, environmental health disparities study in California's Central Valley provides an in-depth look at how to do transdisciplinary research. Working with researchers from the fields of social sciences, public health, biological engineering, and land, air, and water resources, this study aims to answer community residents' questions related to the health disparities they face due to environmental exposure. Through this case study, I articulate not only the logistics of how to conduct transdisciplinary research but also the logics. The implications for transdisciplinary methodologies in health disparity research are further discussed, particularly in the context of team science and convergence science