MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use

Large language models (LLMs) have garnered significant attention due to their impressive natural language processing (NLP) capabilities. Recently, many studies have focused on the tool utilization ability of LLMs. They primarily investigated how LLMs effectively collaborate with given specific tools. However, in scenarios where LLMs serve as intelligent agents, as seen in applications like AutoGPT and MetaGPT, LLMs are expected to engage in intricate decision-making processes that involve deciding whether to employ a tool and selecting the most suitable tool(s) from a collection of available tools to fulfill user requests. Therefore, in this paper, we introduce MetaTool, a benchmark designed to evaluate whether LLMs have tool usage awareness and can correctly choose tools. Specifically, we create a dataset called ToolE within the benchmark. This dataset contains various types of user queries in the form of prompts that trigger LLMs to use tools, including both single-tool and multi-tool scenarios. Subsequently, we set the tasks for both tool usage awareness and tool selection. We define four subtasks from different perspectives in tool selection, including tool selection with similar choices, tool selection in specific scenarios, tool selection with possible reliability issues, and multi-tool selection. We conduct experiments involving nine popular LLMs and find that the majority of them still struggle to effectively select tools, highlighting the existing gaps between LLMs and genuine intelligent agents. However, through the error analysis, we found there is still significant room for improvement. Finally, we conclude with insights for tool developers that follow ChatGPT to provide detailed descriptions that can enhance the tool selection performance of LLMs

Evaluating the efficiency of a nomogram based on the data of neurosurgical intensive care unit patients to predict pulmonary infection of multidrug-resistant Acinetobacter baumannii

BackgroundPulmonary infection caused by multidrug-resistant Acinetobacter baumannii (MDR-AB) is a common and serious complication after brain injury. There are no definitive methods for its prediction and it is usually accompanied by a poor prognosis. This study aimed to construct and evaluate a nomogram based on patient data from the neurosurgical intensive care unit (NSICU) to predict the probability of MDR-AB pulmonary infection.MethodsIn this study, we retrospectively collected patient clinical profiles, early laboratory test results, and doctors’ prescriptions (66 variables). Univariate and backward stepwise regression analyses were used to screen the variables to identify predictors, and a nomogram was built in the primary cohort based on the results of a logistic regression model. Discriminatory validity, calibration validity, and clinical utility were evaluated using validation cohort 1 based on receiver operating characteristic curves, calibration curves, and decision curve analysis (DCA). For external validation based on predictors, we prospectively collected information from patients as validation cohort 2.ResultsAmong 2115 patients admitted to the NSICU between December 1, 2019, and December 31, 2021, 217 were eligible for the study, including 102 patients with MDR-AB infections (102 cases) and 115 patients with other bacterial infections (115 cases). We randomly categorized the patients into the primary cohort (70%, N=152) and validation cohort 1 (30%, N=65). Validation cohort 2 consisted of 24 patients admitted to the NSICU between January 1, 2022, and March 31, 2022, whose clinical information was prospectively collected according to predictors. The nomogram, consisting of only six predictors (age, NSICU stay, Glasgow Coma Scale, meropenem, neutrophil to lymphocyte ratio, platelet to lymphocyte ratio), had significantly high sensitivity and specificity (primary cohort AUC=0.913, validation cohort 1 AUC=0.830, validation cohort 2 AUC=0.889) for early identification of infection and had great calibration (validation cohort 1,2 P=0.3801, 0.6274). DCA confirmed that the nomogram is clinically useful.ConclusionOur nomogram could help clinicians make early predictions regarding the onset of pulmonary infection caused by MDR-AB and implement targeted interventions

Rain Rendering and Construction of <i>Rain Vehicle Color</i>-24 Dataset

The fine identification of vehicle color can assist in criminal investigation or intelligent traffic management law enforcement. Since almost all vehicle-color datasets that are used to train models are collected in good weather, the existing vehicle-color recognition algorithms typically show poor performance for outdoor visual tasks. In this paper we construct a new RainVehicleColor-24 dataset by rain-image rendering using PS technology and a SyRaGAN algorithm based on the VehicleColor-24 dataset. The dataset contains a total of 40,300 rain images with 125 different rain patterns, which can be used to train deep neural networks for specific vehicle-color recognition tasks. Experiments show that the vehicle-color recognition algorithms trained on the new dataset RainVehicleColor-24 improve accuracy to around 72% and 90% on rainy and sunny days, respectively. The code is available at [email protected]

Synthesis, Crystal Structure and Anti-Fatigue Effects of Some Benzamide Derivatives

A series of benzamide derivatives such as 1-(1,3-benzodioxol-5-ylcarbonyl) piperidine (1-BCP) were synthesized by the reaction of substituted benzoic acids with piperidine, morpholine or pyrrolidine using a novel method. The crystals of these benzamide derivatives were obtained by recrystallization. Structures of target and intermediate compounds were determined via FT-IR, 1H-NMR and elemental analysis and X-ray crystallography of select examples. The crystal structures of these compounds have potential applications to identify the binding site for allosteric modulators of the α-amino-3-hydroxy-5-methylisoxazole-4-propionic acid (AMPA) receptor. The anti-fatigue effects of the benzamide derivatives in weight-loaded forced swimming mice were investigated in a swimming endurance capacity test used as an indicator of fatigue. The swimming times to exhaustion were longer in the b3, d3, and e3 groups than in the caffeine group (p &lt; 0.05). In conclusion, b3, d3 and e3 enhanced the forced swimming capacity of mice. The mechanism of the anti-fatigue effects will be studied in the future

Synthesis, Crystal Structure and Anti-Fatigue Effects of Some Benzamide Derivatives

A series of benzamide derivatives such as 1-(1,3-benzodioxol-5-ylcarbonyl) piperidine (1-BCP) were synthesized by the reaction of substituted benzoic acids with piperidine, morpholine or pyrrolidine using a novel method. The crystals of these benzamide derivatives were obtained by recrystallization. Structures of target and intermediate compounds were determined via FT-IR, 1H-NMR and elemental analysis and X-ray crystallography of select examples. The crystal structures of these compounds have potential applications to identify the binding site for allosteric modulators of the α-amino-3-hydroxy-5-methylisoxazole-4-propionic acid (AMPA) receptor. The anti-fatigue effects of the benzamide derivatives in weight-loaded forced swimming mice were investigated in a swimming endurance capacity test used as an indicator of fatigue. The swimming times to exhaustion were longer in the b3, d3, and e3 groups than in the caffeine group (p &lt; 0.05). In conclusion, b3, d3 and e3 enhanced the forced swimming capacity of mice. The mechanism of the anti-fatigue effects will be studied in the future

The IL-17 pathway intertwines with neurotrophin and TLR/IL-1R pathways since its domain shuffling origin

International audienceThe IL-17 pathway displays remarkably diverse functional modes between different subphyla, classes, and even orders, yet its driving factors remains elusive. Here, we demonstrate that the IL-17 pathway originated through domain shuffling between a Toll-like receptor (TLR)/IL-1R pathway and a neurotrophin-RTK (receptor-tyrosine-kinase) pathway (a Trunk-Torso pathway). Unlike other new pathways that evolve independently, the IL-17 pathway remains intertwined with its donor pathways throughout later evolution. This intertwining not only influenced the gains and losses of domains and components in the pathway but also drove the diversification of the pathway’s functional modes among animal lineages. For instance, we reveal that the crustacean female sex hormone, a neurotrophin inducing sex differentiation, could interact with IL-17Rs and thus be classified as true IL-17s. Additionally, the insect prothoracicotropic hormone, a neurotrophin initiating ecdysis in Drosophila by binding to Torso, could bind to IL-17Rs in other insects. Furthermore, IL-17R and TLR/IL-1R pathways maintain crosstalk in amphioxus and zebrafish. Moreover, the loss of the Death domain in the pathway adaptor connection to IκB kinase and stress-activated protein kinase (CIKSs) dramatically reduced their abilities to activate nuclear factor-kappaB (NF-κB) and activator protein 1 (AP-1) in amphioxus and zebrafish. Reinstating this Death domain not only enhanced NF-κB/AP-1 activation but also strengthened anti-bacterial immunity in zebrafish larvae. This could explain why the mammalian IL-17 pathway, whose CIKS also lacks Death, is considered a weak signaling activator, relying on synergies with other pathways. Our findings provide insights into the functional diversity of the IL-17 pathway and unveil evolutionary principles that could govern the pathway and be used to redesign and manipulate it

Association of neutrophil to lymphocyte ratio and D-dimer with functional outcome in patients with cerebral venous sinus thrombosis

Abstract Background Investigations on the risk factors for the prognosis of cerebral venous sinus thrombosis (CVST) are limited. This study aimed to explore whether specific inflammatory factors and coagulation indictors are associated with functional outcome in patients treated for CVST. Methods This retrospective study included 137 patients admitted to our hospital between January 2010 and October 2021. The functional outcome was assessed with the modified Rankin Scale (mRS) score at discharge. Patients were divided into two groups, 102 patients with favorable outcomes (mRS 0-1) and 35 patients with poor outcomes (mRS 2-6). The clinical indexes were compared between two groups. Multivariable logistic regression was performed to identify the independent influencing factors for poor outcomes of CVST patients. The prognostic indicators were analyzed using the receiver operating characteristic (ROC) curve. Results Compared with the favorable outcome group, the incidence of impaired consciousness and brain lesion, the levels of D-dimer, RDW, neutrophil count, neutrophil to lymphocyte ratio (NLR) and red blood cell distribution width to platelet ratio (%) on admission were significantly higher in the poor outcome group, while the level of lymphocyte count was significantly lower. After multivariable logistic regression analysis, baseline D-dimer level (odds ratio (OR), 1.180; 95% confidence interval (CI), 1.019-1.366, P = 0.027) and NLR (OR, 1.903; 95%CI, 1.232-2.938, P = 0.004) were significantly associated with unfavorable outcome at discharge. The ROC curve analysis showed that the areas under the curve of D-dimer, NLR and their combined detection for predicting worse outcome were 0.719, 0.707 and 0.786, respectively. Conclusions Elevated D-dimer level and NLR on admission were associated with an increased risk of poor functional outcome in patients with CVST