Machine Learning, Human Factors and Security Analysis for the Remote Command of Driving: An MCity Pilot

Conducted under the U.S. DOT Office of the Assistant Secretary for Research and Technology’s (OST-R) University Transportation Centers (UTC) program.Both human drivers and autonomous vehicles are able to drive relatively well in frequently encountered settings, but fail in exceptional cases. These exceptional cases often arise suddenly, leaving human drivers with a few seconds at best to react—exactly the setting that people perform worst in. Autonomous systems also fail in exceptional cases, because ambiguous situations preceding crashes are not effectively captured in training datasets. This work introduces new methods for leveraging groups of people to provide on-demand assistance by coordinating responses and using collective answer distributions to generate responses to ambiguous scenarios using minimal time and effort. Unlike prior approaches, we introduce collective workflows that enable groups of people to significantly outperform any of the constituent individuals in terms of time and accuracy. First, we examine the latency and accuracy of crowd workers in a future state prediction task in visual driving scenes, and find that more than 50% of workers could provide accurate answers within one second. We found that using crowd predictions is a viable approach for determining critical future states to inform rapid decision making. Additionally, we characterize different estimation techniques that can be used to efficiently create collective answer distributions from crowd workers for visual tasks containing ambiguity. Surprisingly, we discovered that the most fine-grained and time-consuming methods were not the most accurate. Instead, having annotators choose all relevant responses they thought other annotators would select led to more accurate aggregate outcomes. This approach reduced human time required by 21.4% while maintaining the same level of accuracy as the baseline approach. These research results can inform the development of hybrid intelligence systems that accurately and rapidly address sudden and rare critical events, even when they are ambiguous or subjective.United States Department of Transportation Office of the Assistant Secretary for Research and TechnologyCenter for Connected and Automated Transportationhttp://deepblue.lib.umich.edu/bitstream/2027.42/156392/4/Machine Learning Human Factors and Security Analysis for the Remote Command of Driving - An Mcity Pilot.pd

Public Participation GIS for sustainable urban mobility planning: methods, applications and challenges

Sustainable mobility planning is a new approach to planning, and as such it requires new methods of public participation, data collection and data aggregation. In the article we present an overview of Public Participation GIS (PPGIS) methods with potential use in sustainable urban mobility planning. We present the methods using examples from two recent case studies conducted in Polish cities of Poznań and Łodź. Sustainable urban mobility planning is a cyclical process, and each stage has different data and participatory requirements. Consequently, we situate the PPGIS methods in appropriate stages of planning, based on potential benefits they may bring into the planning process. We discuss key issues related to participant recruitment and provide guidelines for planners interested in implementing methods presented in the paper. The article outlines future research directions stressing the need for systematic case study evaluation

Confidence Contours: Uncertainty-Aware Annotation for Medical Semantic Segmentation

Medical image segmentation modeling is a high-stakes task where understanding of uncertainty is crucial for addressing visual ambiguity. Prior work has developed segmentation models utilizing probabilistic or generative mechanisms to infer uncertainty from labels where annotators draw a singular boundary. However, as these annotations cannot represent an individual annotator's uncertainty, models trained on them produce uncertainty maps that are difficult to interpret. We propose a novel segmentation representation, Confidence Contours, which uses high- and low-confidence ``contours'' to capture uncertainty directly, and develop a novel annotation system for collecting contours. We conduct an evaluation on the Lung Image Dataset Consortium (LIDC) and a synthetic dataset. From an annotation study with 30 participants, results show that Confidence Contours provide high representative capacity without considerably higher annotator effort. We also find that general-purpose segmentation models can learn Confidence Contours at the same performance level as standard singular annotations. Finally, from interviews with 5 medical experts, we find that Confidence Contour maps are more interpretable than Bayesian maps due to representation of structural uncertainty.Comment: 10 pages content, 12 pages total. Accepted to HCOMP '2

Proposed application of the Bayesian Truth Serum for policy analysis

Thesis (S.M. in Technology and Policy)--Massachusetts Institute of Technology, Engineering Systems Division, Technology and Policy Program, 2009.Includes bibliographical references (p. 59-63).Uncertainty and risk are obstacles that nearly all policy-makers encounter during their careers. However, evaluating uncertainty and risk can be difficult since these concepts may be broadly defined. This may result in inaccurate estimates of risk and uncertainty. Expert elicitation is a formal, structured method of obtaining subjective expert judgment in scenarios where objective data is unobtainable. It is designed to reduce the influence of ambiguity on expert judgment, meaning that analysts may use such subjective data as if it were objectively generated. Expert elicitation methods tend to aggregate expert judgment in order to create a unified response, but determining how to combine expert opinions remains a difficult problem. In this thesis, a review of the literature and background behind defining expertise and expert elicitation will be provided. Additionally, this thesis introduces the Bayesian Truth Serum as a potential weighting algorithm for combining expert judgments. As opposed to other weighting algorithms, the Bayesian Truth Serum uses the metaknowledge of experts to create weights for aggregation. Using such information may prove superior to assuming a normal distribution of expertise or relying upon experts to provide estimates of their own expertise.by Rebecca Weiss.S.M.in Technology and Polic

Changing the focus: worker-centric optimization in human-in-the-loop computations

A myriad of emerging applications from simple to complex ones involve human cognizance in the computation loop. Using the wisdom of human workers, researchers have solved a variety of problems, termed as “micro-tasks” such as, captcha recognition, sentiment analysis, image categorization, query processing, as well as “complex tasks” that are often collaborative, such as, classifying craters on planetary surfaces, discovering new galaxies (Galaxyzoo), performing text translation. The current view of “humans-in-the-loop” tends to see humans as machines, robots, or low-level agents used or exploited in the service of broader computation goals. This dissertation is developed to shift the focus back to humans, and study different data analytics problems, by recognizing characteristics of the human workers, and how to incorporate those in a principled fashion inside the computation loop. The first contribution of this dissertation is to propose an optimization framework and a real world system to personalize worker’s behavior by developing a worker model and using that to better understand and estimate task completion time. The framework judiciously frames questions and solicits worker feedback on those to update the worker model. Next, improving workers skills through peer interaction during collaborative task completion is studied. A suite of optimization problems are identified in that context considering collaborativeness between the members as it plays a major role in peer learning. Finally, “diversified” sequence of work sessions for human workers is designed to improve worker satisfaction and engagement while completing tasks