CAME: Contrastive Automated Model Evaluation

The Automated Model Evaluation (AutoEval) framework entertains the possibility of evaluating a trained machine learning model without resorting to a labeled testing set. Despite the promise and some decent results, the existing AutoEval methods heavily rely on computing distribution shifts between the unlabelled testing set and the training set. We believe this reliance on the training set becomes another obstacle in shipping this technology to real-world ML development. In this work, we propose Contrastive Automatic Model Evaluation (CAME), a novel AutoEval framework that is rid of involving training set in the loop. The core idea of CAME bases on a theoretical analysis which bonds the model performance with a contrastive loss. Further, with extensive empirical validation, we manage to set up a predictable relationship between the two, simply by deducing on the unlabeled/unseen testing set. The resulting framework CAME establishes a new SOTA results for AutoEval by surpassing prior work significantly.Comment: ICCV2023 main conferenc

Assessing Hidden Risks of LLMs: An Empirical Study on Robustness, Consistency, and Credibility

The recent popularity of large language models (LLMs) has brought a significant impact to boundless fields, particularly through their open-ended ecosystem such as the APIs, open-sourced models, and plugins. However, with their widespread deployment, there is a general lack of research that thoroughly discusses and analyzes the potential risks concealed. In that case, we intend to conduct a preliminary but pioneering study covering the robustness, consistency, and credibility of LLMs systems. With most of the related literature in the era of LLM uncharted, we propose an automated workflow that copes with an upscaled number of queries/responses. Overall, we conduct over a million queries to the mainstream LLMs including ChatGPT, LLaMA, and OPT. Core to our workflow consists of a data primitive, followed by an automated interpreter that evaluates these LLMs under different adversarial metrical systems. As a result, we draw several, and perhaps unfortunate, conclusions that are quite uncommon from this trendy community. Briefly, they are: (i)-the minor but inevitable error occurrence in the user-generated query input may, by chance, cause the LLM to respond unexpectedly; (ii)-LLMs possess poor consistency when processing semantically similar query input. In addition, as a side finding, we find that ChatGPT is still capable to yield the correct answer even when the input is polluted at an extreme level. While this phenomenon demonstrates the powerful memorization of the LLMs, it raises serious concerns about using such data for LLM-involved evaluation in academic development. To deal with it, we propose a novel index associated with a dataset that roughly decides the feasibility of using such data for LLM-involved evaluation. Extensive empirical studies are tagged to support the aforementioned claims

Training semantic long-term memory retrieval transfers to executive function and reading fluency

The retrieval of information from long-term memory is a fundamental cognitive ability, crucial for most aspects of successful human functioning. Whether and how long-term memory retrieval (LTMR) can be improved with training has clear societal importance but also theoretical value for furthering our understanding of underlying mechanisms. Here, we provide electrophysiological evidence for the plasticity of semantic LTMR. Thirty-five university students were randomly assigned to adaptive semantic LTMR training (using a Posner task) or to a non-adaptive version of the training. Before and after training they were assessed on measures of semantic LTMR, working memory, central executive function (interference control, switching), reading fluency, and fluid intelligence. Adaptive LTMR training (relative to non-adaptive training) led to significant improvements in semantic LTMR. The intervention group (in contrast to the control group) also showed a significant reduction in the mean amplitude of the N400 ERP component and 700–1000 ms measured during a semantic LTMR task, suggesting that changes in retrieval occurred at an early/automatic point and retrieval processing in semantic processing. Moreover, transfer effects were observed for switching, working memory and reading fluency, but not for interference control or fluid intelligence. These results point to the plasticity of semantic LTMR, and suggest that improvement in this ability can transfer to other domains for which LTMR is key

Education in inpatient children and young people’s mental health services

<p>As a chronic disease, osteoarthritis (OA) leads to the degradation of both cartilage and subchondral bone, its development being mediated by proinflammatory cytokines like interleukin-1β. In the present study, the anti-inflammatory effect of specnuezhenide (SPN) in OA and its underlying mechanism were studied in vitro and in vivo. The results showed that SPN decreases the expression of cartilage matrix-degrading enzymes and the activation of NF-κB and wnt/β-catenin signaling, and increases chondrocyte-specific gene expression in IL-1β-induced inflammation in chondrocytes. Furthermore, SPN treatment prevents the degeneration of both cartilage and subchondral bone in a rat model of OA. To the best of our knowledge, this study is the first to report that SPN decreases interleukin-1β-induced inflammation in rat chondrocytes by inhibiting the activation of the NF-κB and wnt/β-catenin pathways, and, thus, has therapeutic potential in the treatment of OA.</p

PHSkb: A knowledgebase to support notifiable disease surveillance

BACKGROUND: Notifiable disease surveillance in the United States is predominantly a passive process that is often limited by poor timeliness and low sensitivity. Interoperable tools are needed that interact more seamlessly with existing clinical and laboratory data to improve notifiable disease surveillance. DESCRIPTION: The Public Health Surveillance Knowledgebase (PHSkb™) is a computer database designed to provide quick, easy access to domain knowledge regarding notifiable diseases and conditions in the United States. The database was developed using Protégé ontology and knowledgebase editing software. Data regarding the notifiable disease domain were collected via a comprehensive review of state health department websites and integrated with other information used to support the National Notifiable Diseases Surveillance System (NNDSS). Domain concepts were harmonized, wherever possible, to existing vocabulary standards. The knowledgebase can be used: 1) as the basis for a controlled vocabulary of reportable conditions needed for data aggregation in public health surveillance systems; 2) to provide queriable domain knowledge for public health surveillance partners; 3) to facilitate more automated case detection and surveillance decision support as a reusable component in an architecture for intelligent clinical, laboratory, and public health surveillance information systems. CONCLUSIONS: The PHSkb provides an extensible, interoperable system architecture component to support notifiable disease surveillance. Further development and testing of this resource is needed

Cox-2 Inhibition Protects against Hypoxia/Reoxygenation-Induced Cardiomyocyte Apoptosis via

The present study explored the potential causal link between ischemia-driven cyclooxygenase-2 (COX-2) expression and enhanced apoptosis during myocardial ischemia/reperfusion (I/R) by using H9C2 cardiomyocytes and primary rat cardiomyocytes subjected to hypoxia/reoxygenation (H/R). The results showed that H/R resulted in higher COX-2 expression than that of controls, which was prevented by pretreatment with Helenalin (NFκB specific inhibitor). Furthermore, pretreatment with NS398 (COX-2 specific inhibitor) significantly attenuated H/R-induced cell injury [lower lactate dehydrogenase (LDH) leakage and enhanced cell viability] and apoptosis (higher Bcl2 expression and lower level of cleaved caspases-3 and TUNEL-positive cells) in cardiomyocytes. The amelioration of posthypoxic apoptotic cell death was paralleled by significant attenuation of H/R-induced increases in proinflammatory cytokines [interleukin 6 (IL6) and tumor necrosis factor (TNFα)] and reactive oxygen species (ROS) production and by higher protein expression of phosphorylated Akt and inducible nitric oxide synthase (iNOS) and enhanced nitric oxide production. Moreover, the application of LY294002 (Akt-specific inhibitor) or 1400W (iNOS-selective inhibitor) cancelled the cellular protective effects of NS398. Findings from the current study suggest that activation of NFκB during cardiomyocyte H/R induces the expression of COX-2 and that higher COX-2 expression during H/R exacerbates cardiomyocyte H/R injury via mechanisms that involve cross talks among inflammation, ROS, and Akt/iNOS/NO signaling

Research on Utilizable Calcium from Calcium Carbide Slag with Different Extractors and Its Effect on CO₂ Mineralization

With the increasing accumulation of alkaline industrial solid waste, the mineralization of CO2 using alkaline industrial solid waste has broad application prospects. Carbide slag is highly alkaline and contains a large amount of calcium elements, making it an excellent material for CO2 mineralization. Our idea was to acquire qualified products and fast kinetics by integrating carbide slag utilization and carbon reduction. The reaction route was divided into two steps: calcium extraction and carbonization. In order to achieve efficient extraction of utilizable calcium, we selected NH4Ac as the extraction agent, which has the advantage of buffer protection and environmental friendliness due to being an acetate radical. The extraction efficiency of utilizable calcium exceeded 90% under the conditions of L/S 20:1 and NH4+/Ca2+ 2:1. In the carbonization process, the crystal forms of CaCO3 synthesized by direct carbonation, acid extraction, and ammonium salt were characterized. The formation mechanism of vaterite in ammonium solution and the influence of impurities (Al3+, Mg2+) on the crystal transformation were revealed. This study provides technical support for using alkaline industrial waste to prepare high-purity vaterite. Therefore, alkaline industrial waste can be efficiently and sustainably utilized through CO2 mineralization