Bag of Tricks for In-Distribution Calibration of Pretrained Transformers

While pre-trained language models (PLMs) have become a de-facto standard promoting the accuracy of text classification tasks, recent studies find that PLMs often predict over-confidently. Although various calibration methods have been proposed, such as ensemble learning and data augmentation, most of the methods have been verified in computer vision benchmarks rather than in PLM-based text classification tasks. In this paper, we present an empirical study on confidence calibration for PLMs, addressing three categories, including confidence penalty losses, data augmentations, and ensemble methods. We find that the ensemble model overfitted to the training set shows sub-par calibration performance and also observe that PLMs trained with confidence penalty loss have a trade-off between calibration and accuracy. Building on these observations, we propose the Calibrated PLM (CALL), a combination of calibration techniques. The CALL complements the drawbacks that may occur when utilizing a calibration method individually and boosts both classification and calibration accuracy. Design choices in CALL's training procedures are extensively studied, and we provide a detailed analysis of how calibration techniques affect the calibration performance of PLMs

Minimax Optimal Bandits for Heavy Tail Rewards

Stochastic multiarmed bandits (stochastic MABs) are a problem of sequential decision-making with noisy rewards, where an agent sequentially chooses actions under unknown reward distributions to minimize cumulative regret. The majority of prior works on stochastic MABs assume that the reward distribution of each action has bounded supports or follows light-tailed distribution, i.e., sub-Gaussian distribution. However, in a variety of decision-making problems, the reward distributions follow a heavy-tailed distribution. In this regard, we consider stochastic MABs with heavy-tailed rewards, whose pth moment is bounded by a constant v(p) for 1 < p <= 2. First, we provide theoretical analysis on sub-optimality of the existing exploration methods for heavy-tailed rewards where it has been proven that existing exploration methods do not guarantee a minimax optimal regret bound. Second, to achieve the minimax optimality under heavy-tailed rewards, we propose a minimax optimal robust upper confidence hound (MR-UCB) by providing tight confidence bound of a p-robust estimator. Furthermore, we also propose a minimax optimal robust adaptively perturbed exploration (MR-APE) which is a randomized version of MR-UCB. In particular, unlike the existing robust exploration methods, both proposed methods have no dependence on v(p). Third, we provide the gap-dependent and independent regret bounds of proposed methods and prove that both methods guarantee the minimax optimal regret bound for a heavy-tailed stochastic MAB problem. The proposed methods are the first algorithm that theoretically guarantees the minimax optimality under heavy-tailed reward settings to the best of our knowledge. Finally, we demonstrate the superiority of the proposed methods in simulation with Pareto and Frichet noises with respect to regrets

Evaluation of conditional treatment effects of adjuvant treatments on patients with synovial sarcoma using Bayesian subgroup analysis

Background The impact of adjuvant chemotherapy or radiation therapy on the survival of patients with synovial sarcoma (SS), which is a rare soft-tissue sarcoma, remains controversial. Bayesian statistical approaches and propensity score matching can be employed to infer treatment effects using observational data. Thus, this study aimed to identify the individual treatment effects of adjuvant therapies on the overall survival of SS patients and recognize subgroups of patients who can benefit from specific treatments using Bayesian subgroup analyses. Methods We analyzed data from patients with SS obtained from the surveillance, epidemiology, and end results (SEER) public database. These data were collected between 1984 and 2014. The treatment effects of chemotherapy and radiation therapy on overall survival were evaluated using propensity score matching. Subgroups that could benefit from radiation therapy or chemotherapy were identified using Bayesian subgroup analyses. Results Based on a stratified Kaplan-Meier curve, chemotherapy exhibited a positive average causal effect on survival in patients with SS, whereas radiation therapy did not. The optimal subgroup for chemotherapy includes the following covariates: older than 20 years, male, large tumor (longest diameter > 5 cm), advanced stage (SEER 3), extremity location, and spindle cell type. The optimal subgroup for radiation therapy includes the following covariates: older than 20 years, male, large tumor (longest diameter > 5 cm), early stage (SEER 1), extremity location, and biphasic type. Conclusion In this study, we identified high-risk patients whose variables include age (age > 20 years), gender, tumor size, tumor location, and poor prognosis without adjuvant treatment. Radiation therapy should be considered in the early stages for high-risk patients with biphasic types. Conversely, chemotherapy should be considered for late-stage high-risk SS patients with spindle cell types

Can We Utilize Pre-trained Language Models within Causal Discovery Algorithms?

Scaling laws have allowed Pre-trained Language Models (PLMs) into the field of causal reasoning. Causal reasoning of PLM relies solely on text-based descriptions, in contrast to causal discovery which aims to determine the causal relationships between variables utilizing data. Recently, there has been current research regarding a method that mimics causal discovery by aggregating the outcomes of repetitive causal reasoning, achieved through specifically designed prompts. It highlights the usefulness of PLMs in discovering cause and effect, which is often limited by a lack of data, especially when dealing with multiple variables. Conversely, the characteristics of PLMs which are that PLMs do not analyze data and they are highly dependent on prompt design leads to a crucial limitation for directly using PLMs in causal discovery. Accordingly, PLM-based causal reasoning deeply depends on the prompt design and carries out the risk of overconfidence and false predictions in determining causal relationships. In this paper, we empirically demonstrate the aforementioned limitations of PLM-based causal reasoning through experiments on physics-inspired synthetic data. Then, we propose a new framework that integrates prior knowledge obtained from PLM with a causal discovery algorithm. This is accomplished by initializing an adjacency matrix for causal discovery and incorporating regularization using prior knowledge. Our proposed framework not only demonstrates improved performance through the integration of PLM and causal discovery but also suggests how to leverage PLM-extracted prior knowledge with existing causal discovery algorithms

Outer Membrane Porin F in E. coli Is Critical for Effective Predation by Bdellovibrio

Bdellovibrio and like organisms (BALOs) are a unique bacterial group that live by predating on other bacteria, consuming them from within to grow and replicate before the progeny come out to complete the life cycle. The mechanisms by which these predators recognize their prey and differentiate them from nonprey bacteria, however, are still not clear. Through genetic knockout and complementation studies in different Escherichia coli strains, we found that Bdellovibrio bacteriovorus strain 109J recognizes outer membrane porin F (OmpF) on the E. coli surface and that the activity of the E. coli EnvZ-OmpR regulatory system significantly impacts predation kinetics. OmpF is not the only signal by which BALOs recognize their prey, however, as B. bacteriovorus could eventually predate on the E. coli ??ompF mutant after prolonged incubation. Furthermore, recognizing OmpF as a prey surface structure was dependent on the prey strain, as knocking out OmpF protein homologues in other prey species, including Escherichia fergusonii, Klebsiella pneumoniae, and Salmonella enterica, did not always reduce the predation rate. Consequently, although OmpF was found to be an important surface component used by Bdellovibrio to efficiently recognize and attack E. coli, future work is needed to determine what other prey surface structures are recognized by these predators