LawBench: Benchmarking Legal Knowledge of Large Language Models

Large language models (LLMs) have demonstrated strong capabilities in various aspects. However, when applying them to the highly specialized, safe-critical legal domain, it is unclear how much legal knowledge they possess and whether they can reliably perform legal-related tasks. To address this gap, we propose a comprehensive evaluation benchmark LawBench. LawBench has been meticulously crafted to have precise assessment of the LLMs' legal capabilities from three cognitive levels: (1) Legal knowledge memorization: whether LLMs can memorize needed legal concepts, articles and facts; (2) Legal knowledge understanding: whether LLMs can comprehend entities, events and relationships within legal text; (3) Legal knowledge applying: whether LLMs can properly utilize their legal knowledge and make necessary reasoning steps to solve realistic legal tasks. LawBench contains 20 diverse tasks covering 5 task types: single-label classification (SLC), multi-label classification (MLC), regression, extraction and generation. We perform extensive evaluations of 51 LLMs on LawBench, including 20 multilingual LLMs, 22 Chinese-oriented LLMs and 9 legal specific LLMs. The results show that GPT-4 remains the best-performing LLM in the legal domain, surpassing the others by a significant margin. While fine-tuning LLMs on legal specific text brings certain improvements, we are still a long way from obtaining usable and reliable LLMs in legal tasks. All data, model predictions and evaluation code are released in https://github.com/open-compass/LawBench/. We hope this benchmark provides in-depth understanding of the LLMs' domain-specified capabilities and speed up the development of LLMs in the legal domain

Mendelian randomization and clinical trial evidence supports TYK2 inhibition as a therapeutic target for autoimmune diseases

Background: To explore the associations of genetically proxied TYK2 inhibition with a wide range of disease outcomes and biomarkers to identify therapeutic repurposing opportunities, adverse effects, and biomarkers of efficacy. Methods: The loss-of-function missense variant rs34536443 in TYK2 gene was used as a genetic instrument to proxy the effect of TYK2 inhibition. A phenome-wide Mendelian randomization (MR) study was conducted to explore the associations of genetically-proxied TYK2 inhibition with 1473 disease outcomes in UK Biobank (N = 339,197). Identified associations were examined for replication in FinnGen (N = 260,405). We further performed tissue -specific gene expression MR, colocalization analyses, and MR with 247 blood biomarkers. A systematic review of randomized controlled trials (RCTs) on TYK2 inhibitor was performed to complement the genetic evidence. Findings: PheWAS-MR found that genetically-proxied TYK2 inhibition was associated with lower risk of a wide range of autoimmune diseases. The associations with hypothyroidism and psoriasis were confirmed in MR analysis of tissue-specific TYK2 gene expression and the associations with systemic lupus erythematosus, psoriasis, and rheumatoid arthritis were observed in colocalization analysis. There were nominal associations of genetically-proxied TYK2 inhibition with increased risk of prostate and breast cancer but not in tissue-specific expression MR or colocalization analyses. Thirty-seven blood biomarkers were associated with the TYK2 loss-of-function mutation. Evidence from RCTs confirmed the effectiveness of TYK2 inhibitors on plaque psoriasis and reported several adverse effects. Interpretation: This study supports TYK2 inhibitor as a potential treatment for psoriasis and several other autoim-mune diseases. Increased pharmacovigilance is warranted in relation to the potential adverse effects.De två första författarna delar förstaförfattarskapet.De tre sista författarna delar sistaförfattarskapet.</p

Molecular Variation and Genomic Function of Citrus Vein Enation Virus

In this study, we identified a new citrus vein enation virus (CVEV) isolate (named CVEV-DT1) through sRNA high-throughput sequencing and traditional sequencing. Phylogenetic analysis based on whole genome sequences of all known CVEV isolates revealed that CVEV-DT1 was in an evolutionary branch with other isolates from China. Molecular variation analysis showed that the single nucleotide variability along CVEV full-length sequences was less than 8%, with more transitions (60.55%) than transversions (39.43%), indicating a genetically homogeneous CVEV population. In addition, non-synonymous nucleotide mutations mainly occurred in ORF1 and ORF2. Based on disorder analysis of all encoded ORF by CVEV-DT1, we identified that the CVEV-DT1 coat protein (CP) formed spherical granules, mainly in the cell nucleus and partly throughout the cytoplasm, with liquid properties through subcellular localization and photobleaching assay. Furthermore, we also confirmed that the CVEV P0 protein has weak post-transcriptional RNA-silencing suppressor activity and could elicit a strong hypersensitive response (HR) in tobacco plants. Collectively, to the best of our knowledge, our study was the first to profile the genomic variation in all the reported CVEV isolates and reveal the functions of CVEV-DT1-encoded proteins