Perspectives on Incorporating Expert Feedback into Model Updates

Machine learning (ML) practitioners are increasingly tasked with developing models that are aligned with non-technical experts' values and goals. However, there has been insufficient consideration on how practitioners should translate domain expertise into ML updates. In this paper, we consider how to capture interactions between practitioners and experts systematically. We devise a taxonomy to match expert feedback types with practitioner updates. A practitioner may receive feedback from an expert at the observation- or domain-level, and convert this feedback into updates to the dataset, loss function, or parameter space. We review existing work from ML and human-computer interaction to describe this feedback-update taxonomy, and highlight the insufficient consideration given to incorporating feedback from non-technical experts. We end with a set of open questions that naturally arise from our proposed taxonomy and subsequent survey

Learning Personalized Decision Support Policies

Individual human decision-makers may benefit from different forms of support to improve decision outcomes. However, a key question is which form of support will lead to accurate decisions at a low cost. In this work, we propose learning a decision support policy that, for a given input, chooses which form of support, if any, to provide. We consider decision-makers for whom we have no prior information and formalize learning their respective policies as a multi-objective optimization problem that trades off accuracy and cost. Using techniques from stochastic contextual bandits, we propose $\texttt{THREAD}$, an online algorithm to personalize a decision support policy for each decision-maker, and devise a hyper-parameter tuning strategy to identify a cost-performance trade-off using simulated human behavior. We provide computational experiments to demonstrate the benefits of $\texttt{THREAD}$ compared to offline baselines. We then introduce $\texttt{Modiste}$, an interactive tool that provides $\texttt{THREAD}$ with an interface. We conduct human subject experiments to show how $\texttt{Modiste}$ learns policies personalized to each decision-maker and discuss the nuances of learning decision support policies online for real users.Comment: Working pape

FeedbackLogs: Recording and Incorporating Stakeholder Feedback into Machine Learning Pipelines

Even though machine learning (ML) pipelines affect an increasing array of stakeholders, there is little work on how input from stakeholders is recorded and incorporated. We propose FeedbackLogs, addenda to existing documentation of ML pipelines, to track the input of multiple stakeholders. Each log records important details about the feedback collection process, the feedback itself, and how the feedback is used to update the ML pipeline. In this paper, we introduce and formalise a process for collecting a FeedbackLog. We also provide concrete use cases where FeedbackLogs can be employed as evidence for algorithmic auditing and as a tool to record updates based on stakeholder feedback

The Fermi GBM Gamma-Ray Burst Spectral Catalog: Four Years Of Data

In this catalog we present the updated set of spectral analyses of GRBs detected by the Fermi Gamma-Ray Burst Monitor (GBM) during its first four years of operation. It contains two types of spectra, time-integrated spectral fits and spectral fits at the brightest time bin, from 943 triggered GRBs. Four different spectral models were fitted to the data, resulting in a compendium of more than 7500 spectra. The analysis was performed similarly, but not identically to Goldstein et al. 2012. All 487 GRBs from the first two years have been re-fitted using the same methodology as that of the 456 GRBs in years three and four. We describe, in detail, our procedure and criteria for the analysis, and present the results in the form of parameter distributions both for the observer-frame and rest-frame quantities. The data files containing the complete results are available from the High-Energy Astrophysics Science Archive Research Center (HEASARC).Comment: Accepted for publication in ApJ

Openness in Education as a Praxis: From Individual Testimonials to Collective Voices

Why is Openness in Education important, and why is it critically needed at this moment? As manifested in our guiding question, the significance of Openness in Education and its immediate necessity form the heart of this collaborative editorial piece. This rather straightforward, yet nuanced query has sparked this collective endeavour by using individual testimonies, which may also be taken as living narratives, to reveal the value of Openness in Education as a praxis. Such testimonies serve as rich, personal narratives, critical introspections, and experience-based accounts that function as sources of data. The data gleaned from these narratives points to the understanding of Openness in Education as a complex, multilayered concept intricately woven into an array of values. These range from aspects such as sharing, access, flexibility, affordability, enlightenment, barrier-removal, empowerment, care, individual agency, trust, innovation, sustainability, collaboration, co-creation, social justice, equity, transparency, inclusivity, decolonization, democratisation, participation, liberty, and respect for diversity. This editorial, as a product of collective endeavour, invites its readers to independently engage with individual narratives, fostering the creation of unique interpretations. This call stems from the distinctive character of each narrative as they voice individual researchers’ perspectives from around the globe, articulating their insights within their unique situational contexts

Openness in Education as a Praxis: From Individual Testimonials to Collective Voices

Why is Openness in Education important, and why is it critically needed at this moment? As manifested in our guiding question, the significance of Openness in Education and its immediate necessity form the heart of this collaborative editorial piece. This rather straightforward, yet nuanced query has sparked this collective endeavour by using individual testimonies, which may also be taken as living narratives, to reveal the value of Openness in Education as a praxis. Such testimonies serve as rich, personal narratives, critical introspections, and experience-based accounts that function as sources of data. The data gleaned from these narratives points to the understanding of Openness in Education as a complex, multilayered concept intricately woven into an array of values. These range from aspects such as sharing, access, flexibility, affordability, enlightenment, barrier-removal, empowerment, care, individual agency, trust, innovation, sustainability, collaboration, co-creation, social justice, equity, transparency, inclusivity, decolonization, democratisation, participation, liberty, and respect for diversity. This editorial, as a product of collective endeavour, invites its readers to independently engage with individual narratives, fostering the creation of unique interpretations. This call stems from the distinctive character of each narrative as they voice individual researchers’ perspectives from around the globe, articulating their insights within their unique situational contexts

Openness in Education as a Praxis: From Individual Testimonials to Collective Voices

Why is Openness in Education important, and why is it critically needed at this moment? As manifested in our guiding question, the significance of Openness in Education and its immediate necessity form the heart of this collaborative editorial piece. This rather straightforward, yet nuanced query has sparked this collective endeavour by using individual testimonies, which may also be taken as living narratives, to reveal the value of Openness in Education as a praxis. Such testimonies serve as rich, personal narratives, critical introspections, and experience-based accounts that function as sources of data. The data gleaned from these narratives points to the understanding of Openness in Education as a complex, multilayered concept intricately woven into an array of values. These range from aspects such as sharing, access, flexibility, affordability, enlightenment, barrier-removal, empowerment, care, individual agency, trust, innovation, sustainability, collaboration, co-creation, social justice, equity, transparency, inclusivity, decolonization, democratisation, participation, liberty, and respect for diversity. This editorial, as a product of collective endeavour, invites its readers to independently engage with individual narratives, fostering the creation of unique interpretations. This call stems from the distinctive character of each narrative as they voice individual researchers’ perspectives from around the globe, articulating their insights within their unique situational contexts