Efficiently Measuring the Cognitive Ability of LLMs: An Adaptive Testing Perspective

Large language models (LLMs), like ChatGPT, have shown some human-like cognitive abilities. For comparing these abilities of different models, several benchmarks (i.e. sets of standard test questions) from different fields (e.g., Literature, Biology and Psychology) are often adopted and the test results under traditional metrics such as accuracy, recall and F1, are reported. However, such way for evaluating LLMs can be inefficient and inaccurate from the cognitive science perspective. Inspired by Computerized Adaptive Testing (CAT) used in psychometrics, we propose an adaptive testing framework for LLM evaluation. Rather than using a standard test set and simply reporting accuracy, this approach dynamically adjusts the characteristics of the test questions, such as difficulty, based on the model's performance. This allows for a more accurate estimation of the model's abilities, using fewer questions. More importantly, it allows LLMs to be compared with humans easily, which is essential for NLP models that aim for human-level ability. Our diagnostic reports have found that ChatGPT often behaves like a ``careless student'', prone to slip and occasionally guessing the questions. We conduct a fine-grained diagnosis and rank the latest 6 instruction-tuned LLMs from three aspects of Subject Knowledge, Mathematical Reasoning, and Programming, where GPT4 can outperform other models significantly and reach the cognitive ability of middle-level students. Different tests for different models using efficient adaptive testing -- we believe this has the potential to become a new norm in evaluating large language models

Quantitative survey of pharmacy students' attitudes and use of traditional Chinese medicine (TCM)

This survey is to study the attitudes, personal experiences of Health care professional students, and knowledge about the basic concepts of Traditional Chinese Medicine (TCM) in Hong Kong and Guangzhou district of Mainland China. The research methodology used was a questionnaire which was distributed amongst healthcare professional students in Hong Kong and Guangzhou. About Seventy-five percent of participants have ever used TCM at some time to treat minor diseases. On a scale from zero to ten, the mean (SD) scores for the perceived effectiveness of TCM were 6.55 (2.07), and most of them (89%) gave five to nine scores. The average proportion of TCM used in their whole lives estimate was 24%. Most of participants (83.3%) used TCM only one to five times per year in the last five years. In participants’ opinion, TCM is more natural and having less adverse effect, while WM is pure, specific, credible, easy to administer and having better patients’ compliance. WM was the priority, or even first choice when the participants faced with a variety of representative diseases. And people had a good understanding degree of some basic concepts of TCM. Although there were a few differences in gender groups comparing and district groups comparing, broadly speaking the similarities still were in majority. In conclusion, most of pharmacy students have their own knowledge about TCM, but more measures must be carried out to improve the popularizing rate of TCM. Moreover, mainland students have better knowledge of TCM and more positive attitude compared to TCM. The attitude and practice do not appear to vary significantly between genders.published_or_final_versionPharmacology and PharmacyMasterMaster of Medical Science

Polymyxin B-induced skin hyperpigmentation: a rare case report and literature review

Abstract Background Polymyxin B (PMB), which is regarded as the ultimate antibacterial treatment against some intractable gram-negative bacteria with its outstanding anti-bacterial activities, inflicts several adverse effects on patients. However, skin hyperpigmentaion (SH) induced by PMB is very rare. Here, we report a case of polymyxin B-induced skin hyperpigmentation (PMB-iSH) in a 21-year-old female. To the best of our knowledge, this is the first case of PMB-iSH in China. Case presentation A 21-year-old female patient with sepsis received the administration of PMB by intravenous injection for the treatment of multi-drug resistant Klebsiella pneumoniae (MDR-KP) infection. She later suffered from a rare adverse drug reaction (ADR), namely PMB-iSH, after 5-day PMB administration during her treatment. There were multiple red rashes spread on the whole body skin at first. With the rashes fading away, SH with dark round spots appeared, associated with no pain or pruritus. The skin of the head and neck was darkened evidently, and dark brown spots were spread on the skin of trunk and limbs. About a month after her admission, urged by the relatives, the patient was transferred back to the local hospital for further treatment in the end, and her skin color didn’t recover to the previous state at that time. Conclusion Both our case and the literature review highlight that PMB can give rise to SH indeed. Clinicians and pharmacists should attach great importance to this rare pigmentary disorder and further investigation is warranted

Acute kidney injury should not be neglected – optimization of quick Pitt bacteremia score for predicting mortality in critically ill patients with bloodstream infection: a retrospective cohort study

Background: Considering the therapeutic difficulties and mortality associated with bloodstream infection (BSI), it is essential to investigate other potential factors affecting mortality in critically ill patients with BSI and examine the utility of the quick Pitt bacteremia (qPitt) score to improve the survival rate. Objectives: To improve the predictive accuracy of the qPitt scoring system by evaluating the five current components of qPitt and including other potential factors influencing mortality in critically ill patients with BSI. Design: This was a retrospective cohort study. Methods: Medical information from the Medical Information Mart for Intensive Care IV database was used in this retrospective cohort study. The risk factors associated with mortality were examined using a multivariate logistic regression model. The area under the receiver operating characteristic curve (AUC) was used to assess the discriminatory capability of the prediction models. Results: In total, 1240 eligible critically ill patients with BSI were included. After adjustment for age, community-onset BSI, indwelling invasive lines, and Glasgow Coma Scale (GCS) ⩽ 8, acute kidney injury (AKI) was identified as a notable risk factor for 14-day mortality. Except for altered mental status, the four other main components of the original qPitt were significantly associated with 14-day mortality. Hence, we established a modified qPitt (m-qPitt) by adding AKI and replacing altered mental status with GCS ⩽ 8. The AUCs for m-qPitt and qPitt were 0.723 [95% confidence interval (CI): 0.683–0.759] and 0.708 (95% CI: 0.669–0.745) in predicting 14-day mortality, respectively. Moreover, m-qPitt also had acceptable performance and discrimination power [0.700 (95% CI: 0.666–0.732)] in predicting 28-day mortality. Conclusion: AKI significantly influenced the survival of critically ill patients with BSIs. Compared with the original qPitt, our new m-qPitt was proven to have a better predictive performance for mortality in critically ill patients with BSI. Further studies should be conducted to validate the practicality of m-qPitt

Identification and Expression Pattern of EZH2 in Pig Developing Fetuses

The proper methylation status of histones is essential for appropriate cell lineage and organogenesis. EZH2, a methyltransferase catalyzing H3K27me3, has been abundantly studied in human and mouse embryonic development. The pig is an increasing important animal model for molecular study and pharmaceutical research. However, the transcript variant and temporal expression pattern of EZH2 in the middle and late porcine fetus are still unknown. Here, we identified the coding sequence of the EZH2 gene and characterized its expression pattern in fetal tissues of Duroc pigs at 65- and 90-day postcoitus (dpc). Our results showed that the coding sequence of EZH2 was 2241 bp, encoding 746 amino acids. There were 9 amino acid insertions and an amino acid substitution in this transcript compared with the validated reference sequence in NCBI. EZH2 was ubiquitously expressed in the fetal tissues of two time points with different expression levels. These results validated a different transcript in pigs and characterized its expression profile in fetal tissues of different gestation stages, which indicated that EZH2 played important roles during porcine embryonic development

Imported monkeypox, Singapore

In May 2019, we investigated monkeypox in a traveler from Nigeria to Singapore. The public health response included rapid identification of contacts, use of quarantine, and postexposure smallpox vaccination. No secondary cases were identified. Countries should develop surveillance systems to detect emerging infectious diseases globally.Published versio