LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models

The advent of large language models (LLMs) and their adoption by the legal community has given rise to the question: what types of legal reasoning can LLMs perform? To enable greater study of this question, we present LegalBench: a collaboratively constructed legal reasoning benchmark consisting of 162 tasks covering six different types of legal reasoning. LegalBench was built through an interdisciplinary process, in which we collected tasks designed and hand-crafted by legal professionals. Because these subject matter experts took a leading role in construction, tasks either measure legal reasoning capabilities that are practically useful, or measure reasoning skills that lawyers find interesting. To enable cross-disciplinary conversations about LLMs in the law, we additionally show how popular legal frameworks for describing legal reasoning—which distinguish between its many forms—correspond to LegalBench tasks, thus giving lawyers and LLM developers a common vocabulary. This paper describes LegalBench, presents an empirical evaluation of 20 open-source and commercial LLMs, and illustrates the types of research explorations LegalBench enables

The added value of a face-to-face pan-European course—what makes it worth it?

IntroductionOver the past decade, digital education has seen widespread adoption, particularly accentuated during the COVID-19 pandemic. The post-COVID era has further emphasized the advantages of digital education in terms of cost, availability, and sustainability. However, concerns regarding the efficacy of digital education, particularly in skills-based learning and the absence of social interaction, have been raised. This paper will look at the added value of international, face-to-face, skills-based courses.MethodThis study evaluates the potential added value of face-to-face international skills courses using the European “Gynecology Experts Training for Upcoming Professionals” (GET-UP) course. Focus group discussions were conducted with participants and faculty members to explore beliefs, attitudes, and perceptions regarding face-to-face learning. Qualitative analysis was performed using thematic analysis to identify domains of added value.ResultsThe GET-UP course, conducted over 4 days with a diverse European faculty and participants, highlighted several added-value domains. Themes including diversity, role models, preparation, live interaction, and community emerged from the analysis, emphasizing the significance of face-to-face interaction in enriching the learning experience beyond attaining learning goals.DiscussionThe study underscores the importance of face-to-face interaction in educational settings, offering insights into diverse teaching methods, role modeling opportunities, enhanced preparation, live interactions, and fostering a sense of community. While digital education continues to evolve with interactive features, this study suggests that the inherent pressure and dynamics of face-to-face learning provide unique benefits that may not be easily replicated in digital environments. Future research should investigate and validate these findings further to inform educational practices effectively