Assessing Human Error Against a Benchmark of Perfection

An increasing number of domains are providing us with detailed trace data on human decisions in settings where we can evaluate the quality of these decisions via an algorithm. Motivated by this development, an emerging line of work has begun to consider whether we can characterize and predict the kinds of decisions where people are likely to make errors. To investigate what a general framework for human error prediction might look like, we focus on a model system with a rich history in the behavioral sciences: the decisions made by chess players as they select moves in a game. We carry out our analysis at a large scale, employing datasets with several million recorded games, and using chess tablebases to acquire a form of ground truth for a subset of chess positions that have been completely solved by computers but remain challenging even for the best players in the world. We organize our analysis around three categories of features that we argue are present in most settings where the analysis of human error is applicable: the skill of the decision-maker, the time available to make the decision, and the inherent difficulty of the decision. We identify rich structure in all three of these categories of features, and find strong evidence that in our domain, features describing the inherent difficulty of an instance are significantly more powerful than features based on skill or time.Comment: KDD 2016; 10 page

Distilling Information Reliability and Source Trustworthiness from Digital Traces

Online knowledge repositories typically rely on their users or dedicated editors to evaluate the reliability of their content. These evaluations can be viewed as noisy measurements of both information reliability and information source trustworthiness. Can we leverage these noisy evaluations, often biased, to distill a robust, unbiased and interpretable measure of both notions? In this paper, we argue that the temporal traces left by these noisy evaluations give cues on the reliability of the information and the trustworthiness of the sources. Then, we propose a temporal point process modeling framework that links these temporal traces to robust, unbiased and interpretable notions of information reliability and source trustworthiness. Furthermore, we develop an efficient convex optimization procedure to learn the parameters of the model from historical traces. Experiments on real-world data gathered from Wikipedia and Stack Overflow show that our modeling framework accurately predicts evaluation events, provides an interpretable measure of information reliability and source trustworthiness, and yields interesting insights about real-world events.Comment: Accepted at 26th World Wide Web conference (WWW-17

Learning Personalized Models of Human Behavior in Chess

Even when machine learning systems surpass human ability in a domain, there are many reasons why AI systems that capture human-like behavior would be desirable: humans may want to learn from them, they may need to collaborate with them, or they may expect them to serve as partners in an extended interaction. Motivated by this goal of human-like AI systems, the problem of predicting human actions -- as opposed to predicting optimal actions -- has become an increasingly useful task. We extend this line of work by developing highly accurate personalized models of human behavior in the context of chess. Chess is a rich domain for exploring these questions, since it combines a set of appealing features: AI systems have achieved superhuman performance but still interact closely with human chess players both as opponents and preparation tools, and there is an enormous amount of recorded data on individual players. Starting with an open-source version of AlphaZero trained on a population of human players, we demonstrate that we can significantly improve prediction of a particular player's moves by applying a series of fine-tuning adjustments. Furthermore, we can accurately perform stylometry -- predicting who made a given set of actions -- indicating that our personalized models capture human decision-making at an individual level.Comment: The current version of the paper corrects data processing problems present in the previous version. 21 pages, 13 figures, 7 tables (one very long

Social media use, online political discussion and UK political events 2013-2018: a phenomenographic study

A thesis submitted to the University of Bedfordshire, in fulfilment of the requirements for the degree of Ph.D.Social media has had observably significant effects on the way many ordinary people participate in politics and appears both symptomatic and causal of a changing landscape. Research, often data-led, has shown marked trends in online behaviour, such as political polarisation, the tendency to form echo chambers and other distinct patterns in the way people debate, share opinions, express their self-identities, consume media and think critically, or otherwise, about political issues. A review of the literature shows that current research in this area across disciplines explores an increasingly wide range of potential influencing factors behind these phenomena, from the social to the psychological to the physiological. However, there have been – far - fewer phenomenological or phenomenographical studies into people’s lived experience of being part of this cultural shift, how their own inclinations, practices and behaviour might be helping to shape the bigger picture, and to what extent they understand this. Starting from an interdisciplinary theoretical framework, and based on in-depth conversations with 84 mostly UK-based adults spoken to one-to-one or in focus groups and webinars over an 18-month period, this study asked people’s about their own perceptions and understanding of their online engagement, focusing on recent major UK political events between 2013 and 2018, (including the Scottish Independence Referendum, The EU Referendum and the Labour Party leadership contests) and considers some of the inferences that might be drawn from people’s own insights. It shows:  People’s experiences are varied, influenced by a range of factors but there is a focus on personal needs and concerns as much as wider political ones  Participants often struggle with behavioural self-awareness and understanding of the motives and actions of others  They can have profound emotional responses owing to the difficulties of using social media but still value it as a medium for political learning and self-expression  A lot of activity takes places in covert, limited or private spaces  Social media itself is an unprecedented learning environment where people begin to understand their own behaviour better and adap