Uncovering Latent Biases in Text: Method and Application to Peer Review

Quantifying systematic disparities in numerical quantities such as employment rates and wages between population subgroups provides compelling evidence for the existence of societal biases. However, biases in the text written for members of different subgroups (such as in recommendation letters for male and non-male candidates), though widely reported anecdotally, remain challenging to quantify. In this work, we introduce a novel framework to quantify bias in text caused by the visibility of subgroup membership indicators. We develop a nonparametric estimation and inference procedure to estimate this bias. We then formalize an identification strategy to causally link the estimated bias to the visibility of subgroup membership indicators, provided observations from time periods both before and after an identity-hiding policy change. We identify an application wherein "ground truth" bias can be inferred to evaluate our framework, instead of relying on synthetic or secondary data. Specifically, we apply our framework to quantify biases in the text of peer reviews from a reputed machine learning conference before and after the conference adopted a double-blind reviewing policy. We show evidence of biases in the review ratings that serves as "ground truth", and show that our proposed framework accurately detects these biases from the review text without having access to the review ratings

CALIBRATING LEGAL JUDGMENTS

Objective: to study the notion and essence of legal judgments calibration, the possibilities of using it in the law-enforcement activity; to explore the expenses and advantages of using it.Methods: dialectic approach to the cognition of social phenomena, which enables to analyze them in historical development and functioning in the context of the integrity of objective and subjective factors; it determined the choice of the following research methods: formal-legal, comparative legal, sociological, methods of cognitive psychology and philosophy.Results: In ordinary life, people who assess other people›s judgments typically take into account the other judgments of those they are assessing in order to calibrate the judgment presently being assessed. The restaurant and hotel rating website TripAdvisor is exemplary, because it facilitates calibration by providing access to a rater›s previous ratings. Such information allows a user to see whether a particular rating comes from a rater who is enthusiastic about every place she patronizes, or instead from someone who is incessantly hard to please. And even when less systematized, as in assessing a letter of recommendation or college transcript, calibration by recourse to the decisional history of those whose judgments are being assessed is ubiquitous. Yet despite the ubiquity and utility of such calibration, the legal system seems perversely to reject it. Appellate courts do not openly adjust their standard of review based on the previous judgments of the judge whose decision they are reviewing, nor do judges in reviewing legislative or administrative decisions, magistrates in evaluating search warrant representations, or jurors in assessing witness perception. In most legal domains, calibration by reference to the prior decisions of the reviewee is invisible, either because it does not exist or because reviewing bodies are unwilling to admit using what they in fact know and employ. Scientific novelty: for the first time, the work substantiates that law is reluctant to take account of the past decisions of the individuals and institutions they are reviewing. By looking only at the particular decision under review, and not calibrating the posture of review on the basis of a history of decisions, reviewing courts and other reviewing institutions embody the particularism that is a large part of the American legal tradition.Practical significance: the main provisions and conclusions of the article can be used in scientific and educational activity when viewing the issues of legal judgments calibration

Assessment of Social Preference in Automotive Market using Generalized Multinomial Logistic Regression

Individual auto market share is always one of the major concerns of any auto manufacturing company. It indicates a lot of things about the company such as profitability, competitiveness, short term and long term development and so on. The focus of this paper is to construct a quantitative model that can precisely formulate the social welfare function of the auto market by relating the auto market share with the utilities of the significant vehicle-purchasing criteria (e.g. reliability, safety, etc.) that concern vehicle buyers. Social welfare function is defined as the additive form of the utility of each criterion considered, it’s a good estimation of the customer preferences. The assessment methods used in this research include random utility theory and B-spline fitted logistic regression model. G-test is applied to select the criteria that is significant to the vehicle market social welfare, pseudo R-squareds are used as the model goodness-of-fit measures and Kendall rank correlation coefficient and Matthews correlation coefficient are applied to validate the assessment model. A case study using the U.S. auto market and vehicles related data collected in years of 2013 and 2014 are conducted to illustrate the assessment process of the social welfare function, and the data from 2015 are used to validate the assessment model.Master of Science in EngineeringIndustrial and Systems Engineering, College of Engineering and Computer ScienceUniversity of Michigan-Dearbornhttp://deepblue.lib.umich.edu/bitstream/2027.42/136069/1/Assessment of Social Preference in Automotive Market using Generalized Multinomial Logistic Regression.pdfDescription of Assessment of Social Preference in Automotive Market using Generalized Multinomial Logistic Regression.pdf : Master of Science in Engineering Thesi

How to Calibrate the Scores of Biased Reviewers by Quadratic Programming

Peer reviewing is the key ingredient of evaluating the quality of scientific work. Based on the review scores assigned by the individual reviewers to the submissions, program committees of conferences and journal editors decide which papers to accept for publication and which to reject. However, some reviewers may be more rigorous than others, they may be biased one way or the other, and they often have highly subjective preferences over the papers they review. Moreover, each reviewer usually has only a very local view, as he or she evaluates only a small fraction of the submissions. Despite all these shortcomings, the review scores obtained need to be aggregrated in order to globally rank all submissions and to make the acceptance/rejection decision. A common method is to simply take the average of each submission's review scores, possibly weighted by the reviewers' confidence levels. Unfortunately, the global ranking thus produced often suffers a certain unfairness, as the reviewers' biases and limitations are not taken into account. We propose a method for calibrating the scores of reviewers that are potentially biased and blindfolded by having only partial information. Our method uses a maximum likelihood estimator, which estimates both the bias of each individual reviewer and the unknown "ideal" score of each submission. This yields a quadratic program whose solution transforms the individual review scores into calibrated, globally comparable scores. We argue why our method results in a fairer and more reasonable global ranking than simply taking the average of scores. To show its usefulness, we test our method empirically using real-world data