LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models

The advent of large language models (LLMs) and their adoption by the legal community has given rise to the question: what types of legal reasoning can LLMs perform? To enable greater study of this question, we present LegalBench: a collaboratively constructed legal reasoning benchmark consisting of 162 tasks covering six different types of legal reasoning. LegalBench was built through an interdisciplinary process, in which we collected tasks designed and hand-crafted by legal professionals. Because these subject matter experts took a leading role in construction, tasks either measure legal reasoning capabilities that are practically useful, or measure reasoning skills that lawyers find interesting. To enable cross-disciplinary conversations about LLMs in the law, we additionally show how popular legal frameworks for describing legal reasoning—which distinguish between its many forms—correspond to LegalBench tasks, thus giving lawyers and LLM developers a common vocabulary. This paper describes LegalBench, presents an empirical evaluation of 20 open-source and commercial LLMs, and illustrates the types of research explorations LegalBench enables

Experimental Philosophical Bioethics

There is a rich tradition in bioethics of gathering empirical data to inform, supplement, or test the implications of normative ethical analysis. To this end, bioethicists have drawn on diverse methods, including qualitative interviews, focus groups, ethnographic studies, and opinion surveys to advance understanding of key issues in bioethics. In so doing, they have developed strong ties with neighboring disciplines such as anthropology, history, law, and sociology. Collectively, these lines of research have flourished in the broader field of “empirical bioethics” for more than 30 years (Sugarman & Sulmasy 2010). More recently, philosophers from outside the field of bioethics have similarly employed empirical methods—drawn primarily from psychology, the cognitive sciences, economics, and related disciplines—to advance theoretical debates. This approach, which has come to be called experimental philosophy (or x-phi), relies primarily on controlled experiments to interrogate the concepts, intuitions, reasoning, implicit mental processes, and empirical assumptions about the mind that play a role in traditional philosophical arguments (Knobe et al. 2012). Within the moral domain, for example, experimental philosophy has begun to contribute to long-standing debates about the nature of moral judgment and reasoning; the sources of our moral emotions and biases; the qualities of a good person or a good life; and the psychological basis of moral theory itself (Alfano, Loeb, & Plakias 2018). We believe that experimental philosophical bioethics—or “bioxphi”—can similarly explain how it is distinct from empirical bioethics more broadly construed, and attempt to characterize how it might advance theory and practice in this area

Addressing climate change with behavioral science: a global intervention tournament in 63 countries

Effectively reducing climate change requires marked, global behavior change. However, it is unclear which strategies are most likely to motivate people to change their climate beliefs and behaviors. Here, we tested 11 expert-crowdsourced interventions on four climate mitigation outcomes: beliefs, policy support, information sharing intention, and an effortful tree-planting behavioral task. Across 59,440 participants from 63 countries, the interventions’ effectiveness was small, largely limited to nonclimate skeptics, and differed across outcomes: Beliefs were strengthened mostly by decreasing psychological distance (by 2.3%), policy support by writing a letter to a future-generation member (2.6%), information sharing by negative emotion induction (12.1%), and no intervention increased the more effortful behavior—several interventions even reduced tree planting. Last, the effects of each intervention differed depending on people’s initial climate beliefs. These findings suggest that the impact of behavioral climate interventions varies across audiences and target behaviors

Addressing climate change with behavioral science::A global intervention tournament in 63 countries

Effectively reducing climate change requires marked, global behavior change. However, it is unclear which strategies are most likely to motivate people to change their climate beliefs and behaviors. Here, we tested 11 expert-crowdsourced interventions on four climate mitigation outcomes: beliefs, policy support, information sharing intention, and an effortful tree-planting behavioral task. Across 59,440 participants from 63 countries, the interventions’ effectiveness was small, largely limited to nonclimate skeptics, and differed across outcomes: Beliefs were strengthened mostly by decreasing psychological distance (by 2.3%), policy support by writing a letter to a future-generation member (2.6%), information sharing by negative emotion induction (12.1%), and no intervention increased the more effortful behavior—several interventions even reduced tree planting. Last, the effects of each intervention differed depending on people’s initial climate beliefs. These findings suggest that the impact of behavioral climate interventions varies across audiences and target behaviors

Addressing climate change with behavioral science: A global intervention tournament in 63 countries

International audienceEffectively reducing climate change requires marked, global behavior change. However, it is unclear which strategies are most likely to motivate people to change their climate beliefs and behaviors. Here, we tested 11 expert-crowdsourced interventions on four climate mitigation outcomes: beliefs, policy support, information sharing intention, and an effortful tree-planting behavioral task. Across 59,440 participants from 63 countries, the interventions’ effectiveness was small, largely limited to nonclimate skeptics, and differed across outcomes: Beliefs were strengthened mostly by decreasing psychological distance (by 2.3%), policy support by writing a letter to a future-generation member (2.6%), information sharing by negative emotion induction (12.1%), and no intervention increased the more effortful behavior—several interventions even reduced tree planting. Last, the effects of each intervention differed depending on people’s initial climate beliefs. These findings suggest that the impact of behavioral climate interventions varies across audiences and target behaviors