Reinforcement Learning from Statistical Feedback: the Journey from AB Testing to ANT Testing

Reinforcement Learning from Human Feedback (RLHF) has played a crucial role in the success of large models such as ChatGPT. RLHF is a reinforcement learning framework which combines human feedback to improve learning effectiveness and performance. However, obtaining preferences feedback manually is quite expensive in commercial applications. Some statistical commercial indicators are usually more valuable and always ignored in RLHF. There exists a gap between commercial target and model training. In our research, we will attempt to fill this gap with statistical business feedback instead of human feedback, using AB testing which is a well-established statistical method. Reinforcement Learning from Statistical Feedback (RLSF) based on AB testing is proposed. Statistical inference methods are used to obtain preferences for training the reward network, which fine-tunes the pre-trained model in reinforcement learning framework, achieving greater business value. Furthermore, we extend AB testing with double selections at a single time-point to ANT testing with multiple selections at different feedback time points. Moreover, we design numerical experiences to validate the effectiveness of our algorithm framework

Effects of acidification on nitrification and associated nitrous oxide emission in estuarine and coastal waters

In the context of an increasing atmospheric carbon dioxide (CO2) level, acidification of estuarine and coastal waters is greatly exacerbated by land-derived nutrient inputs, coastal upwelling, and complex biogeochemical processes. A deeper understanding of how nitrifiers respond to intensifying acidification is thus crucial to predict the response of estuarine and coastal ecosystems and their contribution to global climate change. Here, we show that acidification can significantly decrease nitrification rate but stimulate generation of byproduct nitrous oxide (N2O) in estuarine and coastal waters. By varying CO2 concentration and pH independently, an expected beneficial effect of elevated CO2 on activity of nitrifiers (“CO2-fertilization” effect) is excluded under acidification. Metatranscriptome data further demonstrate that nitrifiers could significantly up-regulate gene expressions associated with intracellular pH homeostasis to cope with acidification stress. This study highlights the molecular underpinnings of acidification effects on nitrification and associated greenhouse gas N2O emission, and helps predict the response and evolution of estuarine and coastal ecosystems under climate change and human activities.publishedVersio

Longitudinal Genomic Evolution of Conventional Papillary Thyroid Cancer With Brain Metastasis

BackgroundBrain metastasis is extremely rare but predicts dismal prognosis in papillary thyroid cancer (PTC). Dynamic evaluation of stepwise metastatic lesions was barely conducted to identify the longitudinal genomic evolution of brain metastasis in PTC.MethodChronologically resected specimen was analyzed by whole exome sequencing, including four metastatic lymph nodes (lyn 1–4) and brain metastasis lesion (BM). Phylogenetic tree was reconstructed to infer the metastatic pattern and the potential functional mutations.ResultsContrasting with lyn1, ipsilateral metastatic lesions (lyn2–4 and BM) with shared biallelic mutations of TSC2 indicated different genetic originations from multifocal tumors. Lyn 3/4, particularly lyn4 exhibited high genetic similarity with BM. Besides the similar mutational compositions and signatures, shared functional mutations (CDK4R24C, TP53R342*) were observed in lyn3/4 and BM. Frequencies of these mutations gradually increase along with the metastasis progression. Consistently, TP53 knockout and CDK4R24C introduction in PTC cells significantly decreased radioiodine uptake and increased metastatic ability.ConclusionGenomic mutations in CDK4 and TP53 during the tumor evolution may contribute to the lymph node and brain metastasis of PTC

Potential of Core-Collapse Supernova Neutrino Detection at JUNO

JUNO is an underground neutrino observatory under construction in Jiangmen, China. It uses 20kton liquid scintillator as target, which enables it to detect supernova burst neutrinos of a large statistics for the next galactic core-collapse supernova (CCSN) and also pre-supernova neutrinos from the nearby CCSN progenitors. All flavors of supernova burst neutrinos can be detected by JUNO via several interaction channels, including inverse beta decay, elastic scattering on electron and proton, interactions on C12 nuclei, etc. This retains the possibility for JUNO to reconstruct the energy spectra of supernova burst neutrinos of all flavors. The real time monitoring systems based on FPGA and DAQ are under development in JUNO, which allow prompt alert and trigger-less data acquisition of CCSN events. The alert performances of both monitoring systems have been thoroughly studied using simulations. Moreover, once a CCSN is tagged, the system can give fast characterizations, such as directionality and light curve

Detection of the Diffuse Supernova Neutrino Background with JUNO

As an underground multi-purpose neutrino detector with 20 kton liquid scintillator, Jiangmen Underground Neutrino Observatory (JUNO) is competitive with and complementary to the water-Cherenkov detectors on the search for the diffuse supernova neutrino background (DSNB). Typical supernova models predict 2-4 events per year within the optimal observation window in the JUNO detector. The dominant background is from the neutral-current (NC) interaction of atmospheric neutrinos with 12C nuclei, which surpasses the DSNB by more than one order of magnitude. We evaluated the systematic uncertainty of NC background from the spread of a variety of data-driven models and further developed a method to determine NC background within 15\% with {\it{in}} {\it{situ}} measurements after ten years of running. Besides, the NC-like backgrounds can be effectively suppressed by the intrinsic pulse-shape discrimination (PSD) capabilities of liquid scintillators. In this talk, I will present in detail the improvements on NC background uncertainty evaluation, PSD discriminator development, and finally, the potential of DSNB sensitivity in JUNO

Real-time Monitoring for the Next Core-Collapse Supernova in JUNO

Core-collapse supernova (CCSN) is one of the most energetic astrophysical events in the Universe. The early and prompt detection of neutrinos before (pre-SN) and during the SN burst is a unique opportunity to realize the multi-messenger observation of the CCSN events. In this work, we describe the monitoring concept and present the sensitivity of the system to the pre-SN and SN neutrinos at the Jiangmen Underground Neutrino Observatory (JUNO), which is a 20 kton liquid scintillator detector under construction in South China. The real-time monitoring system is designed with both the prompt monitors on the electronic board and online monitors at the data acquisition stage, in order to ensure both the alert speed and alert coverage of progenitor stars. By assuming a false alert rate of 1 per year, this monitoring system can be sensitive to the pre-SN neutrinos up to the distance of about 1.6 (0.9) kpc and SN neutrinos up to about 370 (360) kpc for a progenitor mass of 30$M_{\odot}$ for the case of normal (inverted) mass ordering. The pointing ability of the CCSN is evaluated by using the accumulated event anisotropy of the inverse beta decay interactions from pre-SN or SN neutrinos, which, along with the early alert, can play important roles for the followup multi-messenger observations of the next Galactic or nearby extragalactic CCSN.Comment: 24 pages, 9 figure

Influence of stratospheric sudden warming on the tropical intraseasonal convection

Madden–Julian oscillation (MJO), the dominant mode of intraseasonal variability in the tropical troposphere, has recently been shown to have a great impact on Northern Hemisphere (NH) extratropical stratosphere. But the influence of the variability in the extratropical stratosphere on MJO is seldom reported. In this study, the influence of major, mid–winter NH stratospheric sudden warmings (SSWs) on the MJO is investigated using meteorological reanalysis datasets. Our analysis reveals that SSWs also exert considerable influence on tropical intraseasonal convection. The occurrences of MJO phases 6 and 7 significantly increase during around 20 d after the onset of SSWs, corresponding to enhanced convective activity over the equatorial Central and Western Pacific. Then in the following days, the coherent eastward propagation of tropical intraseasonal convection resembles the periodic variation in a typical MJO. These results suggest that the extratropical stratosphere affects the organized tropical intraseasonal convection, and variability of the tropical intraseasonal convection related to MJO can be better grasped by taking extratropical stratospheric variability into account. Considering the complex interaction between MJO and extratropical stratosphere, further work on comprehensive understanding of the relationship between SSWs and MJO is required in future studies

Container Ship Carbon and Fuel Estimation in Voyages Utilizing Meteorological Data with Data Fusion and Machine Learning Techniques

The International Maritime Organization (IMO) had made effort to reduce the ship’s energy consumption and carbon emission by optimizing the ship’s operational measures such as speed and weather routing. However, existing fuel consumption models were relatively simple without considering the quantified effect of weather conditions. In this paper, a knowledge-based ridge regression-based algorithm is presented for enabling automated fuel consumption estimation under varying weather conditions during voyages. Wind speed, wave height, ship speed, draught, AIS segment distance, and ship’s heading (HDG) are used as input to predict the fuel consumption value from the MRV report. In this work, 3 types of models are tested: AIS-based model, MRV-based model, and MRV-based normalized model. In AIS based model, weather conditions are divided into nine categories based on wind speed, wave height, and wind directions then trained separately. In MRV-based mode, the daily weather condition was used, and the MRV-normalized model used the normalized daily weather data. The proposed ridge regression models (11 models total) were tested with 4 container ships for a period of one year, and the result shows that compared to real fuel consumption, MRV-based model could achieve the best result with an average error less than 3% comparing to real MRV report

Recent Strategies to Address Hypoxic Tumor Environments in Photodynamic Therapy

Photodynamic therapy (PDT) has become a promising method of cancer treatment due to its unique properties, such as noninvasiveness and low toxicity. The efficacy of PDT is, however, significantly reduced by the hypoxia tumor environments, because PDT involves the generation of reactive oxygen species (ROS), which requires the great consumption of oxygen. Moreover, the consumption of oxygen caused by PDT would further exacerbate the hypoxia condition, which leads to angiogenesis, invasion of tumors to other parts, and metastasis. Therefore, many research studies have been conducted to design nanoplatforms that can alleviate tumor hypoxia and enhance PDT. Herein, the recent progress on strategies for overcoming tumor hypoxia is reviewed, including the direct transport of oxygen to the tumor site by O2 carriers, the in situ generation of oxygen by decomposition of oxygen-containing compounds, reduced O2 consumption, as well as the regulation of tumor microenvironments. Limitations and future perspectives of these technologies to improve PDT are also discussed

Understanding social media beyond text: A reliable practice on Twitter

Social media provides high-volume and real-time data, which has been broadly used in diverse applications in sales, marketing, disaster management, health surveillance, etc. However, distinguishing between noises and reliable information can be challenging, since social media, a user-generated content system, has a great number of users who update massive information every second. The rich information is not only included in the short textual content but also embedded in the images and videos. In this paper, we introduce an effective and efficient framework for event detection with social media data. The framework integrates both textual and imagery content in the hope to fully utilize the information. The approach has been demonstrated to be more accurate than the text-only approach by removing 58 (66.7%) false-positive events. The precision of event detection is improved by 6.5%. Besides, based on our analysis, we also look into the content of these images to further explore the space of social media studies. Finally, the closely related text and image from social media offer us a valuable text-image mapping, which can enable knowledge transfer between two media types