A study on a mixed stopping strategy for total recall tasks

How do we calculate how many relevant documents are in a collection? In this abstract, we discuss our line of research about total recall systems such as interactive system for systematic reviews based on an active learning framework [4\u20136]. In particular, we will present 1) the problem in mathematical terms, and 2) the experiments of an interactive system that continuously monitors the costs of reviewing additional documents and suggests the user whether to continue or not in the search based on the available remaining resources. We will discuss the results of this system on the ongoing CLEF 2019 eHealth task

Predictive Coding Techniques with Manual Review to Identify Privileged Documents in E-Discovery

In twenty-first century civil litigation, discovery focuses on the retrieval of electronically stored information. Lawsuits may be won or lost because of incorrect production of electronic evidence. Organizations may generate fewer paper documents, leading to an increase in the amount of electronic documents by many fold. Litigants face the task of searching millions of electronic records for the presence of responsive and not-privileged documents, making the e-discovery process burdensome and expensive. In order to ensure that the material that has to be withheld is not inadvertently revealed, the electronic evidence that is found to be responsive to a production request is typically subjected to an exhaustive manual review for privilege. Although the budgetary constraints on review for responsiveness can be met using automation to some degree, attorneys have been hesitant to adopt similar technology to support the privilege review process. This dissertation draws attention to the potential for adopting predictive coding technology for the privilege review phase during the discovery process. Two main questions that are central to building a privilege classifier are addressed. The first question seeks to determine which set of annotations can serve as a reliable basis for evaluation. The second question seeks to determine which of the remaining annotations, when used for training classifiers, produce the best results. As an answer, binary classifiers are trained on labeled annotations from both junior and senior reviewers. Issues related to training bias and sample variance due to the reviewer's expertise are thoroughly discussed. Results show that the annotations that were randomly drawn and annotated by senior reviewers are useful for evaluation. The remaining annotations can be used for classifier training. A research prototype is built to perform a user study. Privilege judgments are gathered from multiple lawyers using two user interfaces. One of the two interfaces includes automatically generated features to aid the review process. The goal is to help lawyers make faster and more accurate privilege judgments. A significant improvement in recall was noted when comparing the users' review performance when using the automated annotations. Classifier features related to the people involved in privileged communications were found to be particularly important for the privilege review task. Results show that there was no measurable change in review time. As cost is proportional to time during review, as the final step, this work introduces a semi-automated framework that aims to optimize the cost of the manual review process. The framework calls for litigants to make some rational choices about what to manually review. The documents are first automatically classified for responsiveness and privilege, and then some of the automatically classified documents are reviewed by human reviewers for responsiveness and for privilege with the overall goal of minimizing the expected cost of the entire process, including costs that arise from incorrect decisions. A risk-based ranking algorithm is used to determine which documents need to be manually reviewed. Multiple baselines are used to characterize the cost savings achieved by this approach. Although the work in this dissertation is applied to e-discovery, similar approaches could be applied to any case in which retrieval systems have to withhold a set of confidential documents despite their relevance to the request

More on the Ethics of E-Discovery: Predictive Coding and Other Forms of Computer-Assisted Review

This paper was circulated at a TAR conference hosted by the Bolch Judicial Institute (then the Center for Judicial Studies) in 2015. With the author\u27s permission, the paper has been archived in the scholarship repository. This document does not represent the views of Duke Law School, Duke University, their faculties, or any other organization

Joint Resolution of Supply Chain Risks: The Role of Risk Characteristics and Problem Solving Approach

The purpose of this study is to examine the disruption risk resolution process in supply chains; specifically, to assess how risk attributes impact the approach firms select to resolve risks and the associated final outcomes. We propose that high magnitude risks are positively associated with mutually beneficial problem resolution; on the other hand, low likelihood risks have the opposite effect, they are negatively associated with mutually beneficial resolution. Our conceptual contribution lies in our articulation of the mechanisms though which risk magnitude and risk likelihood impact mutual problem resolution. We posit that high magnitude risks and low likelihood (uncommon) risks mobilize the social network of actors, triggering vigilant monitoring for risks, communication among actors and across firm boundaries, and resource sharing and coordination which facilitate collaborative problem solving and mutual resolutions. These mobilization mechanisms help supply chain partners to overcome the challenges of complexity and allow for information and resource flows among actors and between firms. Our statistical analysis demonstrates that the impact of risk attributes on mutual problem solutions is fully mediated by timely problem identification and collaborative problem solving

Corporate Law Practice as a Public Calling

Law is a service profession; but it is also a public profession. Lawyers are supposed to serve their clients faithfully and zealously; but they also are supposed to work, both on and off the job of representing clients, as counselors, citizens, reformers, community activists, and public servants, to maintain the integrity of the framework of laws, institutions, and procedures that constrain their clients\u27 practices and their own-and not just to maintain that framework, but to help transform it so that it more nearly will approach the conditions ofjustice and civic community. The lawyer is to function, in Talcott Parsons\u27 words, as a kind of buffer between the illegitimate desires of his client and the social interest. He represents the client before the legal system; but he also represents the legal system to the client

Selling State Borders

Sovereign territory was bought and sold throughout much of American history, and there are good reasons to think that an interstate market for borders could help solve many contemporary economic and political problems. But no such market currently exists. Why not? And could an interstate market for sovereign territory help simplify border disputes, resolve state budget crises, respond to exogenous shocks like river accretion, and improve democratic responsiveness? Focusing on the sale of borders among American states, this Article offers constitutional, political, and ethical answers to the first question, and a qualified yes to the second