PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization

Instruction tuning large language models (LLMs) remains a challenging task, owing to the complexity of hyperparameter selection and the difficulty involved in evaluating the tuned models. To determine the optimal hyperparameters, an automatic, robust, and reliable evaluation benchmark is essential. However, establishing such a benchmark is not a trivial task due to the challenges associated with evaluation accuracy and privacy protection. In response to these challenges, we introduce a judge large language model, named PandaLM, which is trained to distinguish the superior model given several LLMs. PandaLM's focus extends beyond just the objective correctness of responses, which is the main focus of traditional evaluation datasets. It addresses vital subjective factors such as relative conciseness, clarity, adherence to instructions, comprehensiveness, and formality. To ensure the reliability of PandaLM, we collect a diverse human-annotated test dataset, where all contexts are generated by humans and labels are aligned with human preferences. Our results indicate that PandaLM-7B achieves 93.75% of GPT-3.5's evaluation ability and 88.28% of GPT-4's in terms of F1-score on our test dataset. PandaLM enables the evaluation of LLM to be fairer but with less cost, evidenced by significant improvements achieved by models tuned through PandaLM compared to their counterparts trained with default Alpaca's hyperparameters. In addition, PandaLM does not depend on API-based evaluations, thus avoiding potential data leakage. All resources of PandaLM are released at https://github.com/WeOpenML/PandaLM

Plasma microRNAs as potential biomarkers for non-small-cell lung cancer

Non-small-cell lung cancer (NSCLC) is the leading cause of cancer-related death. Developing minimally invasive techniques that can diagnose NSCLC, particularly at an early stage, may improve its outcome. Using microarray platforms, we previously identified 12 microRNAs (miRNAs) the aberrant expressions of which in primary lung tumors are associated with early-stage NSCLC. Here, we extend our previous research by investigating whether the miRNAs could be used as potential plasma biomarkers for NSCLC. We initially validated expressions of the miRNAs in paired lung tumor tissues and plasma specimens from 28 stage I NSCLC patients by real-time quantitative reverse transcription PCR, and then evaluated diagnostic value of the plasma miRNAs in a cohort of 58 NSCLC patients and 29 healthy individuals. The altered miRNA expressions were reproducibly confirmed in the tumor tissues. The miRNAs were stably present and reliably measurable in plasma. Of the 12 miRNAs, five displayed significant concordance of the expression levels in plasma and the corresponding tumor tissues (all r>0.850, all P<0.05). A logistic regression model with the best prediction was defined on the basis of the four genes (miRNA-21, -126, -210, and 486-5p), yielding 86.22% sensitivity and 96.55% specificity in distinguishing NSCLC patients from the healthy controls. Furthermore, the panel of miRNAs produced 73.33% sensitivity and 96.55% specificity in identifying stage I NSCLC patients. In addition, the genes have higher sensitivity (91.67%) in diagnosis of lung adenocarcinomas compared with squamous cell carcinomas (82.35%) (P<0.05). Altered expressions of the miRNAs in plasma would provide potential blood-based biomarkers for NSCLC

Layer-dependent anisotropic frictional behavior in two-dimensional monolayer hybrid perovskite/ITO layered heterojunctions

Two-dimensional (2D) organic-inorganic hybrid perovskites, which possess outstanding optical and electrical properties, are promising semiconductor materials that have attracted significant interest in widespread applications. The frictional behavior of 2D perovskite materials with other transparent conductive materials, such as indium tin oxide (ITO), offers promising developments in optoelectronic devices. Therefore, the understanding of this frictional behavior is essential. Atomic force microscopy (AFM) is employed here to measure the frictional behavior between the (001) plane of the 2D organic-inorganic hybrid (C₄H₉NH₃)₂PbBr₄ perovskite and the (111) plane of the ITO. The experimental analyses characterizing the nature of the friction in a single-crystalline heterojunction are reported. Based on the results of the analyses of interfaces between 2D monolayer perovskites and ITO, a strong anisotropy of friction is clearly demonstrated. The anisotropy of friction is observed as a four-fold symmetry with low a frictional coefficient, 0.035, in misaligned contacts, and, 0.015, in aligned contacts in the heterojunction configuration. In addition, atomistic simulations reveal underlying frictional mechanisms in the dynamical regimes. A new phenomenon discovered in the studies establishes that the measured frictional anisotropy surprisingly depends on the number of atomic layers in the 2D perovskite. The frictional anisotropy decreases significantly with the increase in the number of layers up to 16 layers, and then it becomes independent of the thickness. Our results are predicted to be of a general nature and should be applicable to other 2D hybrid perovskite heterojunction configurations, and thus, furthers the development of adaptive and stretchable optoelectronic nanodevices.This project was financially supported by the National Natural Science Foundation of China (NSFC, 51702035, 51605079 and 51602056), and Dalian University of Technology, China, (DUT16RC(3)051)