Fairness Hub Technical Briefs: AUC Gap

To measure bias, we encourage teams to consider using AUC Gap: the absolute difference between the highest and lowest test AUC for subgroups (e.g., gender, race, SES, prior knowledge). It is agnostic to the AI/ML algorithm used and it captures the disparity in model performance for any number of subgroups, which enables non-binary fairness assessments such as for intersectional identity groups. The teams use a wide range of AI/ML models in pursuit of a common goal of doubling math achievement in low-income middle schools. Ensuring that the models, which are trained on datasets collected in many different contexts, do not introduce or amplify biases is important for achieving the goal. We offer here a versatile and easy-to-compute measure of model bias for all the teams in order to create a common benchmark and an analytical basis for sharing what strategies have worked for different teams

Rethinking the Evaluation Protocol of Domain Generalization

Domain generalization aims to solve the challenge of Out-of-Distribution (OOD) generalization by leveraging common knowledge learned from multiple training domains to generalize to unseen test domains. To accurately evaluate the OOD generalization ability, it is necessary to ensure that test data information is unavailable. However, the current domain generalization protocol may still have potential test data information leakage. This paper examines the potential risks of test data information leakage in two aspects of the current protocol: pretraining on ImageNet and oracle model selection. We propose that training from scratch and using multiple test domains would result in a more precise evaluation of OOD generalization ability. We also rerun the algorithms with the modified protocol and introduce a new leaderboard to encourage future research in domain generalization with a fairer comparison

Flatness-Aware Minimization for Domain Generalization

Domain generalization (DG) seeks to learn robust models that generalize well under unknown distribution shifts. As a critical aspect of DG, optimizer selection has not been explored in depth. Currently, most DG methods follow the widely used benchmark, DomainBed, and utilize Adam as the default optimizer for all datasets. However, we reveal that Adam is not necessarily the optimal choice for the majority of current DG methods and datasets. Based on the perspective of loss landscape flatness, we propose a novel approach, Flatness-Aware Minimization for Domain Generalization (FAD), which can efficiently optimize both zeroth-order and first-order flatness simultaneously for DG. We provide theoretical analyses of the FAD's out-of-distribution (OOD) generalization error and convergence. Our experimental results demonstrate the superiority of FAD on various DG datasets. Additionally, we confirm that FAD is capable of discovering flatter optima in comparison to other zeroth-order and first-order flatness-aware optimization methods.Comment: Accepted by ICCV202

Expression profiles of microRNAs in skeletal muscle of sheep by deep sequencing

Objective MicroRNAs are a class of endogenous small regulatory RNAs that regulate cell proliferation, differentiation and apoptosis. Recent studies on miRNAs are mainly focused on mice, human and pig. However, the studies on miRNAs in skeletal muscle of sheep are not comprehensive. Methods RNA-seq technology was used to perform genomic analysis of miRNAs in prenatal and postnatal skeletal muscle of sheep. Targeted genes were predicted using miRanda software and miRNA-mRNA interactions were verified by quantitative real-time polymerase chain reaction. To further investigate the function of miRNAs, candidate targeted genes were enriched for analysis using gene ontology (GO) and Kyoto encyclopedia of genes and genomes (KEGG) enrichment. Results The results showed total of 1,086 known miRNAs and 40 new candidate miRNAs were detected in prenatal and postnatal skeletal muscle of sheep. In addition, 345 miRNAs (151 up-regulated, 94 down-regulated) were differentially expressed. Moreover, miRanda software was performed to predict targeted genes of miRNAs, resulting in a total of 2,833 predicted targets, especially miR-381 which targeted multiple muscle-related mRNAs. Furthermore, GO and KEGG pathway analysis confirmed that targeted genes of miRNAs were involved in development of skeletal muscles. Conclusion This study supplements the miRNA database of sheep, which provides valuable information for further study of the biological function of miRNAs in sheep skeletal muscle

Equity-Oriented Educational Data Science

As educational institutions increasingly adopt digital tools for daily operations, unprecedented amounts of data are generated at different levels of the education system. The granularity of these big data makes it possible to understand and support educational processes in a data-informed, easy-to-scale manner, and educational data science (EDS) has emerged as a nascent field to realize this potential. This dissertation specifically focuses on the promise of EDS to address issues related to educational equity, a central theme of education research. To begin with, a two-dimensional taxonomy is presented to characterize equity-oriented EDS research -- whether the work is focused on explanation or prediction, and whether the problem of interest takes a micro- or macro-level perspective of education research. The interaction of these two dimensions partitions the research space into four quadrants, and one empirical study in higher education contexts is presented to illustrate each quadrant. The first two explanatory studies leverage novel data sources (i.e., digital behavioral traces) to understand systematic sociodemographic gaps in 1) peer interaction experience in virtual learning environments (micro-level), and 2) academic engagement over time at the institutional level (macro-level). The latter two prediction-oriented studies investigate algorithmic fairness from the perspectives of 1) choice of data sources in online learning performance prediction (micro-level), and 2) use of sensitive attributes in early warning systems (macro-level). These studies highlight how EDS research can advance the understanding of existing educational inequalities and guide preventive action against potential inequities. Finally, recommendations for future research on equity-oriented EDS are discussed

Model-Based Offline Weighted Policy Optimization (Student Abstract)

A promising direction for applying reinforcement learning to the real world is learning from offline datasets. Offline reinforcement learning aims to learn policies from pre-collected datasets without online interaction with the environment. Due to the lack of further interaction, offline reinforcement learning faces severe extrapolation error, leading to policy learning failure. In this paper, we investigate the weighted Bellman update in model-based offline reinforcement learning. We explore uncertainty estimation in ensemble dynamics models, then use a variational autoencoder to fit the behavioral prior, and finally propose an algorithm called Model-Based Offline Weighted Policy Optimization (MOWPO), which uses a combination of model confidence and behavioral prior as weights to reduce the impact of inaccurate samples on policy optimization. Experiment results show that MOWPO achieves better performance than state-of-the-art algorithms, and both the model confidence weight and the behavioral prior weight can play an active role in offline policy optimization