Expert-Augmented Machine Learning

Machine Learning is proving invaluable across disciplines. However, its success is often limited by the quality and quantity of available data, while its adoption by the level of trust that models afford users. Human vs. machine performance is commonly compared empirically to decide whether a certain task should be performed by a computer or an expert. In reality, the optimal learning strategy may involve combining the complementary strengths of man and machine. Here we present Expert-Augmented Machine Learning (EAML), an automated method that guides the extraction of expert knowledge and its integration into machine-learned models. We use a large dataset of intensive care patient data to predict mortality and show that we can extract expert knowledge using an online platform, help reveal hidden confounders, improve generalizability on a different population and learn using less data. EAML presents a novel framework for high performance and dependable machine learning in critical applications

Listening between the Lines: Learning Personal Attributes from Conversations

Open-domain dialogue agents must be able to converse about many topics while incorporating knowledge about the user into the conversation. In this work we address the acquisition of such knowledge, for personalization in downstream Web applications, by extracting personal attributes from conversations. This problem is more challenging than the established task of information extraction from scientific publications or Wikipedia articles, because dialogues often give merely implicit cues about the speaker. We propose methods for inferring personal attributes, such as profession, age or family status, from conversations using deep learning. Specifically, we propose several Hidden Attribute Models, which are neural networks leveraging attention mechanisms and embeddings. Our methods are trained on a per-predicate basis to output rankings of object values for a given subject-predicate combination (e.g., ranking the doctor and nurse professions high when speakers talk about patients, emergency rooms, etc). Experiments with various conversational texts including Reddit discussions, movie scripts and a collection of crowdsourced personal dialogues demonstrate the viability of our methods and their superior performance compared to state-of-the-art baselines.Comment: published in WWW'1

Integrating multiple criteria decision analysis in participatory forest planning

Forest planning in a participatory context often involves multiple stakeholders with conflicting interests. A promising approach for handling these complex situations is to integrate participatory planning and multiple criteria decision analysis (MCDA). The objective of this paper is to analyze strengths and weaknesses of such an integrated approach, focusing on how the use of MCDA has influenced the participatory process. The paper outlines a model for a participatory MCDA process with five steps: stakeholder analysis, structuring of the decision problem, generation of alternatives, elicitation of preferences, and ranking of alternatives. This model was applied in a case study of a planning process for the urban forest in Lycksele, Sweden. In interviews with stakeholders, criteria for four different social groups were identified. Stakeholders also identified specific areas important to them and explained what activities the areas were used for and the forest management they wished for there. Existing forest data were combined with information from interviews to create a map in which the urban forest was divided into zones of different management classes. Three alternative strategic forest plans were produced based on the zonal map. The stakeholders stated their preferences individually by the Analytic Hierarchy Process in inquiry forms and a ranking of alternatives and consistency ratios were determined for each stakeholder. Rankings of alternatives were aggregated; first, for each social group using the arithmetic mean, and then an overall aggregated ranking was calculated from the group rankings using the weighted arithmetic mean. The participatory MCDA process in Lycksele is assessed against five social goals: incorporating public values into decisions, improving the substantive quality of decisions, resolving conflict among competing interests, building trust in institutions, and educating and informing the public. The results and assessment of the case study support the integration of participatory planning and MCDA as a viable option for handling complex forest-management situations. Key issues related to the MCDA methodology that need to be explored further were identified: 1) The handling of place-specific criteria, 2) development of alternatives, 3) the aggregation of individual preferences into a common preference, and 4) application and evaluation of the integrated approach in real case studies

Advancing Alternative Analysis: Integration of Decision Science.

Decision analysis-a systematic approach to solving complex problems-offers tools and frameworks to support decision making that are increasingly being applied to environmental challenges. Alternatives analysis is a method used in regulation and product design to identify, compare, and evaluate the safety and viability of potential substitutes for hazardous chemicals.Assess whether decision science may assist the alternatives analysis decision maker in comparing alternatives across a range of metrics.A workshop was convened that included representatives from government, academia, business, and civil society and included experts in toxicology, decision science, alternatives assessment, engineering, and law and policy. Participants were divided into two groups and prompted with targeted questions. Throughout the workshop, the groups periodically came together in plenary sessions to reflect on other groups' findings.We conclude the further incorporation of decision science into alternatives analysis would advance the ability of companies and regulators to select alternatives to harmful ingredients, and would also advance the science of decision analysis.We advance four recommendations: (1) engaging the systematic development and evaluation of decision approaches and tools; (2) using case studies to advance the integration of decision analysis into alternatives analysis; (3) supporting transdisciplinary research; and (4) supporting education and outreach efforts

Expert-augmented machine learning.

Machine learning is proving invaluable across disciplines. However, its success is often limited by the quality and quantity of available data, while its adoption is limited by the level of trust afforded by given models. Human vs. machine performance is commonly compared empirically to decide whether a certain task should be performed by a computer or an expert. In reality, the optimal learning strategy may involve combining the complementary strengths of humans and machines. Here, we present expert-augmented machine learning (EAML), an automated method that guides the extraction of expert knowledge and its integration into machine-learned models. We used a large dataset of intensive-care patient data to derive 126 decision rules that predict hospital mortality. Using an online platform, we asked 15 clinicians to assess the relative risk of the subpopulation defined by each rule compared to the total sample. We compared the clinician-assessed risk to the empirical risk and found that, while clinicians agreed with the data in most cases, there were notable exceptions where they overestimated or underestimated the true risk. Studying the rules with greatest disagreement, we identified problems with the training data, including one miscoded variable and one hidden confounder. Filtering the rules based on the extent of disagreement between clinician-assessed risk and empirical risk, we improved performance on out-of-sample data and were able to train with less data. EAML provides a platform for automated creation of problem-specific priors, which help build robust and dependable machine-learning models in critical applications