Development of the PEBL Traveling Salesman Problem Computerized Testbed

The traveling salesman problem (TSP) is a combinatorial optimization problem that requires finding the shortest path through a set of points (“cities”) that returns to the starting point. Because humans provide heuristic near-optimal solutions to Euclidean versions of the problem, it has sometimes been used to investigate human visual problem solving ability. The TSP is also similar to a number of tasks commonly used for neuropsychological assessment (such as the trail-making test), and so its utility in assessing reliable individual differences in problem solving has sometimes been examined. Nevertheless, the task has seen little widespread use in clinical and assessment domains, in part because no standard software implementation or item set is widely available with known psychometric properties. In this paper, we describe a computerized version of TSP running in the free and open source Psychology Experiment Building Language (PEBL). The PEBL TSP task is designed to be suitable for use within a larger battery of tests, and to examine both standard and custom TSP node configurations (i.e., problems). We report the results of a series of experiments that help establish the test’s reliability and validity. The first experiment examines test-retest reliability, establishes that the quality of solutions in the TSP are not impacted by mild physiological strain, and demonstrates how solution quality obtained by individuals in a physical version is highly correlated with solution quality obtained in the PEBL version. The second experiment evaluates a larger set of problems, and uses the data to identify a small subset of tests that have maximal coherence. A third experiment examines test-retest reliability of this smaller set that can be administered in about five minutes, and establishes that these problems produce composite scores with moderately high (R = .75) test-retest reliability, making it suitable for use in many assessment situations, including evaluations of individual differences, personality, and intelligence testing

Decision noise may mask criterion shifts: Reply to Balakrishnan and MacDonald (2008)

J. D. Balakrishnan and J. A. MacDonald (2008) argue that RTbased measures of signal detection processes provide evidence against signal detection theory’s notion of a flexible decision criterion. They argue that this evidence is immune to the alternative explanation proposed by S. T. Mueller and C. T. Weidemann (2008), that decision noise may mask criterion shifts. We show that noise in response times can produce the same effects as are produced by noise in confidence ratings. Given these results, the evidence is not sufficient to categorically reject the notion of a flexible response policy implemented through shifts in a decision criterion

Explainable AI: roles and stakeholders, desirements and challenges

IntroductionThe purpose of the Stakeholder Playbook is to enable the developers of explainable AI systems to take into account the different ways in which different stakeholders or role-holders need to “look inside” the AI/XAI systems.MethodWe conducted structured cognitive interviews with senior and mid-career professionals who had direct experience either developing or using AI and/or autonomous systems.ResultsThe results show that role-holders need access to others (e.g., trusted engineers and trusted vendors) for them to be able to develop satisfying mental models of AI systems. They need to know how it fails and misleads as much as they need to know how it works. Some stakeholders need to develop an understanding that enables them to explain the AI to someone else and not just satisfy their own sense-making requirements. Only about half of our interviewees said they always wanted explanations or even needed better explanations than the ones that were provided. Based on our empirical evidence, we created a “Playbook” that lists explanation desires, explanation challenges, and explanation cautions for a variety of stakeholder groups and roles.DiscussionThis and other findings seem surprising, if not paradoxical, but they can be resolved by acknowledging that different role-holders have differing skill sets and have different sense-making desires. Individuals often serve in multiple roles and, therefore, can have different immediate goals. The goal of the Playbook is to help XAI developers by guiding the development process and creating explanations that support the different roles

Alphabetic Letter Identification: Effects of perceivability, similarity, and bias

The legibility of the letters in the Latin alphabet has been measured numerous times since the beginning of\ud experimental psychology. To identify the theoretical mechanisms attributed to letter identification, we report\ud a comprehensive review of literature, spanning more than a century. This review revealed that identification\ud accuracy has frequently been attributed to a subset of three common sources: perceivability, bias, and simi-\ud larity. However, simultaneous estimates of these values have rarely (if ever) been performed. We present the\ud results of two new experiments which allow for the simultaneous estimation of these factors, and examine\ud how the shape of a visual mask impacts each of them, as inferred through a new statistical model. Results showed that the shape and identity of the mask impacted the inferred perceivability, bias, and similarity space of a letter set, but that there were aspects of similarity that were robust to the choice of mask. The results illustrate how the psychological concepts of perceivability, bias, and similarity can be estimated simultaneously, and how each make powerful contributions to visual letter identification