26 research outputs found
Pre-registration for Predictive Modeling
Amid rising concerns of reproducibility and generalizability in predictive
modeling, we explore the possibility and potential benefits of introducing
pre-registration to the field. Despite notable advancements in predictive
modeling, spanning core machine learning tasks to various scientific
applications, challenges such as overlooked contextual factors, data-dependent
decision-making, and unintentional re-use of test data have raised questions
about the integrity of results. To address these issues, we propose adapting
pre-registration practices from explanatory modeling to predictive modeling. We
discuss current best practices in predictive modeling and their limitations,
introduce a lightweight pre-registration template, and present a qualitative
study with machine learning researchers to gain insight into the effectiveness
of pre-registration in preventing biased estimates and promoting more reliable
research outcomes. We conclude by exploring the scope of problems that
pre-registration can address in predictive modeling and acknowledging its
limitations within this context
Modality and uncertainty in data visualizations : A corpus approach to the use of connecting lines
publishedVersionPaid Open Acces
REFORMS: Reporting Standards for Machine Learning Based Science
Machine learning (ML) methods are proliferating in scientific research.
However, the adoption of these methods has been accompanied by failures of
validity, reproducibility, and generalizability. These failures can hinder
scientific progress, lead to false consensus around invalid claims, and
undermine the credibility of ML-based science. ML methods are often applied and
fail in similar ways across disciplines. Motivated by this observation, our
goal is to provide clear reporting standards for ML-based science. Drawing from
an extensive review of past literature, we present the REFORMS checklist
(porting Standards achine Learning
Based cience). It consists of 32 questions and a paired set of
guidelines. REFORMS was developed based on a consensus of 19 researchers across
computer science, data science, mathematics, social sciences, and biomedical
sciences. REFORMS can serve as a resource for researchers when designing and
implementing a study, for referees when reviewing papers, and for journals when
enforcing standards for transparency and reproducibility
Recommended from our members
Structure and Empathy in Visual Data Storytelling: Evaluating their Influence on Attitude
In the visualization community, it is often assumed that visual data storytelling increases memorability and engagement, making it more effective at communicating information. However, many assumptions about the efficacy of storytelling in visualization lack empirical evaluation. Contributing to an emerging body of work, we study whether selected techniques commonly used in visual data storytelling influence people's attitudes towards immigration. We compare (a) personal visual narratives designed to generate empathy; (b) structured visual narratives of aggregates of people; and (c) an exploratory visualization without narrative acting as a control condition. We conducted two crowdsourced betweenâsubject studies comparing the three conditions, each with 300 participants. To assess the differences in attitudes between conditions, we adopted established scales from the social sciences used in the European Social Survey (ESS). Although we found some differences between conditions, the effects on people's attitudes are smaller than we expected. Our findings suggest that we need to be more careful when it comes to our expectations about the effects visual data storytelling can have on attitudes. Additional material: https://flowstory.github.io/attitudes/
Towards a Conceptual Model for Data Narratives
International audienc
Modality and uncertainty in data visualizations: A corpus approach to the use of connecting lines
publishedVersionPaid Open Acces