401 research outputs found
ClassActionPrediction: A Challenging Benchmark for Legal Judgment Prediction of Class Action Cases in the US
The research field of Legal Natural Language Processing (NLP) has been very
active recently, with Legal Judgment Prediction (LJP) becoming one of the most
extensively studied tasks. To date, most publicly released LJP datasets
originate from countries with civil law. In this work, we release, for the
first time, a challenging LJP dataset focused on class action cases in the US.
It is the first dataset in the common law system that focuses on the harder and
more realistic task involving the complaints as input instead of the often used
facts summary written by the court. Additionally, we study the difficulty of
the task by collecting expert human predictions, showing that even human
experts can only reach 53% accuracy on this dataset. Our Longformer model
clearly outperforms the human baseline (63%), despite only considering the
first 2,048 tokens. Furthermore, we perform a detailed error analysis and find
that the Longformer model is significantly better calibrated than the human
experts. Finally, we publicly release the dataset and the code used for the
experiments
ChatRule: Mining Logical Rules with Large Language Models for Knowledge Graph Reasoning
Logical rules are essential for uncovering the logical connections between
relations, which could improve the reasoning performance and provide
interpretable results on knowledge graphs (KGs). Although there have been many
efforts to mine meaningful logical rules over KGs, existing methods suffer from
the computationally intensive searches over the rule space and a lack of
scalability for large-scale KGs. Besides, they often ignore the semantics of
relations which is crucial for uncovering logical connections. Recently, large
language models (LLMs) have shown impressive performance in the field of
natural language processing and various applications, owing to their emergent
ability and generalizability. In this paper, we propose a novel framework,
ChatRule, unleashing the power of large language models for mining logical
rules over knowledge graphs. Specifically, the framework is initiated with an
LLM-based rule generator, leveraging both the semantic and structural
information of KGs to prompt LLMs to generate logical rules. To refine the
generated rules, a rule ranking module estimates the rule quality by
incorporating facts from existing KGs. Last, a rule validator harnesses the
reasoning ability of LLMs to validate the logical correctness of ranked rules
through chain-of-thought reasoning. ChatRule is evaluated on four large-scale
KGs, w.r.t. different rule quality metrics and downstream tasks, showing the
effectiveness and scalability of our method.Comment: 11 pages, 4 figure
A Hierarchical Neural Framework for Classification and its Explanation in Large Unstructured Legal Documents
Automatic legal judgment prediction and its explanation suffer from the
problem of long case documents exceeding tens of thousands of words, in
general, and having a non-uniform structure. Predicting judgments from such
documents and extracting their explanation becomes a challenging task, more so
on documents with no structural annotation. We define this problem as "scarce
annotated legal documents" and explore their lack of structural information and
their long lengths with a deep-learning-based classification framework which we
call MESc; "Multi-stage Encoder-based Supervised with-clustering"; for judgment
prediction. We explore the adaptability of LLMs with multi-billion parameters
(GPT-Neo, and GPT-J) to legal texts and their intra-domain(legal) transfer
learning capacity. Alongside this, we compare their performance and
adaptability with MESc and the impact of combining embeddings from their last
layers. For such hierarchical models, we also propose an explanation extraction
algorithm named ORSE; Occlusion sensitivity-based Relevant Sentence Extractor;
based on the input-occlusion sensitivity of the model, to explain the
predictions with the most relevant sentences from the document. We explore
these methods and test their effectiveness with extensive experiments and
ablation studies on legal documents from India, the European Union, and the
United States with the ILDC dataset and a subset of the LexGLUE dataset. MESc
achieves a minimum total performance gain of approximately 2 points over
previous state-of-the-art proposed methods, while ORSE applied on MESc achieves
a total average gain of 50% over the baseline explainability scores
Towards Explainability and Fairness in Swiss Judgement Prediction: Benchmarking on a Multilingual Dataset
The assessment of explainability in Legal Judgement Prediction (LJP) systems
is of paramount importance in building trustworthy and transparent systems,
particularly considering the reliance of these systems on factors that may lack
legal relevance or involve sensitive attributes. This study delves into the
realm of explainability and fairness in LJP models, utilizing Swiss Judgement
Prediction (SJP), the only available multilingual LJP dataset. We curate a
comprehensive collection of rationales that `support' and `oppose' judgement
from legal experts for 108 cases in German, French, and Italian. By employing
an occlusion-based explainability approach, we evaluate the explainability
performance of state-of-the-art monolingual and multilingual BERT-based LJP
models, as well as models developed with techniques such as data augmentation
and cross-lingual transfer, which demonstrated prediction performance
improvement. Notably, our findings reveal that improved prediction performance
does not necessarily correspond to enhanced explainability performance,
underscoring the significance of evaluating models from an explainability
perspective. Additionally, we introduce a novel evaluation framework, Lower
Court Insertion (LCI), which allows us to quantify the influence of lower court
information on model predictions, exposing current models' biases.Comment: Accepted at LREC-COLING 202
A Survey on Legal Question Answering Systems
Many legal professionals think that the explosion of information about local,
regional, national, and international legislation makes their practice more
costly, time-consuming, and even error-prone. The two main reasons for this are
that most legislation is usually unstructured, and the tremendous amount and
pace with which laws are released causes information overload in their daily
tasks. In the case of the legal domain, the research community agrees that a
system allowing to generate automatic responses to legal questions could
substantially impact many practical implications in daily activities. The
degree of usefulness is such that even a semi-automatic solution could
significantly help to reduce the workload to be faced. This is mainly because a
Question Answering system could be able to automatically process a massive
amount of legal resources to answer a question or doubt in seconds, which means
that it could save resources in the form of effort, money, and time to many
professionals in the legal sector. In this work, we quantitatively and
qualitatively survey the solutions that currently exist to meet this challenge.Comment: 57 pages, 1 figure, 10 table
Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence
Recent years have seen a tremendous growth in Artificial Intelligence (AI)-based methodological development in a broad range of domains. In this rapidly evolving field, large number of methods are being reported using machine learning (ML) and Deep Learning (DL) models. Majority of these models are inherently complex and lacks explanations of the decision making process causing these models to be termed as 'Black-Box'. One of the major bottlenecks to adopt such models in mission-critical application domains, such as banking, e-commerce, healthcare, and public services and safety, is the difficulty in interpreting them. Due to the rapid proleferation of these AI models, explaining their learning and decision making process are getting harder which require transparency and easy predictability. Aiming to collate the current state-of-the-art in interpreting the black-box models, this study provides a comprehensive analysis of the explainable AI (XAI) models. To reduce false negative and false positive outcomes of these back-box models, finding flaws in them is still difficult and inefficient. In this paper, the development of XAI is reviewed meticulously through careful selection and analysis of the current state-of-the-art of XAI research. It also provides a comprehensive and in-depth evaluation of the XAI frameworks and their efficacy to serve as a starting point of XAI for applied and theoretical researchers. Towards the end, it highlights emerging and critical issues pertaining to XAI research to showcase major, model-specific trends for better explanation, enhanced transparency, and improved prediction accuracy
xxAI - Beyond Explainable AI
This is an open access book.
Statistical machine learning (ML) has triggered a renaissance of artificial intelligence (AI). While the most successful ML models, including Deep Neural Networks (DNN), have developed better predictivity, they have become increasingly complex, at the expense of human interpretability (correlation vs. causality). The field of explainable AI (xAI) has emerged with the goal of creating tools and models that are both predictive and interpretable and understandable for humans.
Explainable AI is receiving huge interest in the machine learning and AI research communities, across academia, industry, and government, and there is now an excellent opportunity to push towards successful explainable AI applications. This volume will help the research community to accelerate this process, to promote a more systematic use of explainable AI to improve models in diverse applications, and ultimately to better understand how current explainable AI methods need to be improved and what kind of theory of explainable AI is needed.
After overviews of current methods and challenges, the editors include chapters that describe new developments in explainable AI. The contributions are from leading researchers in the field, drawn from both academia and industry, and many of the chapters take a clear interdisciplinary approach to problem-solving. The concepts discussed include explainability, causability, and AI interfaces with humans, and the applications include image processing, natural language, law, fairness, and climate science.https://digitalcommons.unomaha.edu/isqafacbooks/1000/thumbnail.jp
xxAI - Beyond Explainable AI
This is an open access book. Statistical machine learning (ML) has triggered a renaissance of artificial intelligence (AI). While the most successful ML models, including Deep Neural Networks (DNN), have developed better predictivity, they have become increasingly complex, at the expense of human interpretability (correlation vs. causality). The field of explainable AI (xAI) has emerged with the goal of creating tools and models that are both predictive and interpretable and understandable for humans. Explainable AI is receiving huge interest in the machine learning and AI research communities, across academia, industry, and government, and there is now an excellent opportunity to push towards successful explainable AI applications. This volume will help the research community to accelerate this process, to promote a more systematic use of explainable AI to improve models in diverse applications, and ultimately to better understand how current explainable AI methods need to be improved and what kind of theory of explainable AI is needed. After overviews of current methods and challenges, the editors include chapters that describe new developments in explainable AI. The contributions are from leading researchers in the field, drawn from both academia and industry, and many of the chapters take a clear interdisciplinary approach to problem-solving. The concepts discussed include explainability, causability, and AI interfaces with humans, and the applications include image processing, natural language, law, fairness, and climate science
- …