On Cognitive Preferences and the Plausibility of Rule-based Models

It is conventional wisdom in machine learning and data mining that logical models such as rule sets are more interpretable than other models, and that among such rule-based models, simpler models are more interpretable than more complex ones. In this position paper, we question this latter assumption by focusing on one particular aspect of interpretability, namely the plausibility of models. Roughly speaking, we equate the plausibility of a model with the likeliness that a user accepts it as an explanation for a prediction. In particular, we argue that, all other things being equal, longer explanations may be more convincing than shorter ones, and that the predominant bias for shorter models, which is typically necessary for learning powerful discriminative models, may not be suitable when it comes to user acceptance of the learned models. To that end, we first recapitulate evidence for and against this postulate, and then report the results of an evaluation in a crowd-sourcing study based on about 3.000 judgments. The results do not reveal a strong preference for simple rules, whereas we can observe a weak preference for longer rules in some domains. We then relate these results to well-known cognitive biases such as the conjunction fallacy, the representative heuristic, or the recogition heuristic, and investigate their relation to rule length and plausibility.Comment: V4: Another rewrite of section on interpretability to clarify focus on plausibility and relation to interpretability, comprehensibility, and justifiabilit

Designing for designers: Towards the development of accessible ICT products and services using the VERITAS framework

Among key design practices which contribute to the development of inclusive ICT products and services is user testing with people with disabilities. Traditionally, this involves partial or minimal user testing through the usage of standard heuristics, employing external assisting devices, and the direct feedback of impaired users. However, efficiency could be improved if designers could readily analyse the needs of their target audience. The VERITAS framework simulates and systematically analyses how users with various impairments interact with the use of ICT products and services. Findings show that the VERITAS framework is useful to designers, offering an intuitive approach to inclusive design.The work presented in this article forms part of VERITAS, which is funded by the European Commission's 7th Framework Programme (FP7) (grant agreement # 247765 FP7-ICT-2009.7.2)

Towards Design Principles for Data-Driven Decision Making: An Action Design Research Project in the Maritime Industry

Data-driven decision making (DDD) refers to organizational decision-making practices that emphasize the use of data and statistical analysis instead of relying on human judgment only. Various empirical studies provide evidence for the value of DDD, both on individual decision maker level and the organizational level. Yet, the path from data to value is not always an easy one and various organizational and psychological factors mediate and moderate the translation of data-driven insights into better decisions and, subsequently, effective business actions. The current body of academic literature on DDD lacks prescriptive knowledge on how to successfully employ DDD in complex organizational settings. Against this background, this paper reports on an action design research study aimed at designing and implementing IT artifacts for DDD at one of the largest ship engine manufacturers in the world. Our main contribution is a set of design principles highlighting, besides decision quality, the importance of model comprehensibility, domain knowledge, and actionability of results

Comprehension and trust in crises: investigating the impact of machine translation and post-editing

We conducted a survey to understand the impact of machine translation and postediting awareness on comprehension of and trust in messages disseminated to prepare the public for a weather-related crisis, i.e. flooding. The translation direction was English–Italian. Sixty-one participants—all native Italian speakers with different English proficiency levels— answered our survey. Each participant read and evaluated between three and six crisis messages using ratings and openended questions on comprehensibility and trust. The messages were in English and Italian. All the Italian messages had been machine translated and post-edited. Nevertheless, participants were told that only half had been post-edited, so that we could test the impact of post-editing awareness. We could not draw firm conclusions when comparing the scores for trust and comprehensibility assigned to the three types of messages—English, post-edits, and purported raw outputs. However, when scores were triangulated with open-ended answers, stronger patterns were observed, such as the impact of fluency of the translations on their comprehensibility and trustworthiness. We found correlations between comprehensibility and trustworthiness, and identified other factors influencing these aspects, such as the clarity and soundness of the messages. We conclude by outlining implications for crisis preparedness, limitations, and areas for future research

Evaluating Visual Realism in Drawing Areas of Interest on UML Diagrams

Areas of interest (AOIs) are defined as an addition to UML diagrams: groups of elements of system architecture diagrams that share some common property. Some methods have been proposed to automatically draw AOIs on UML diagrams. However, it is not clear how users perceive the results of such methods as compared to human-drawn areas of interest. We present here a process of studying and improving the perceived quality of computer-drawn AOIs. We qualitatively evaluated how users perceive the quality of computer- and human-drawn AOIs, and used these results to improve an existing algorithm for drawing AOIs. Finally, we designed a quantitative comparison for AOI drawings and used it to show that our improved renderings are closer to human drawings than the original rendering algorithm results. The combined user evaluation, algorithmic improvements, and quantitative comparison support our claim of improving the perceived quality of AOIs rendered on UML diagrams.

Rethinking the Agreement in Human Evaluation Tasks

Human evaluations are broadly thought to be more valuable the higher the inter-annotator agreement. In this paper we examine this idea. We will describe our experiments and analysis within the area of Automatic Question Generation. Our experiments show how annotators diverge in language annotation tasks due to a range of ineliminable factors. For this reason, we believe that annotation schemes for natural language generation tasks that are aimed at evaluating language quality need to be treated with great care. In particular, an unchecked focus on reduction of disagreement among annotators runs the danger of creating generation goals that reward output that is more distant from, rather than closer to, natural human-like language. We conclude the paper by suggesting a new approach to the use of the agreement metrics in natural language generation evaluation tasks

Do Process Modelling Techniques Get Better? A Comparative Ontological Analysis of BPMN

Current initiatives in the field of Business Process Management (BPM) strive for the development of a BPM standard notation by pushing the Business Process Modeling Notation (BPMN). However, such a proposed standard notation needs to be carefully examined. Ontological analysis is an established theoretical approach to evaluating modelling techniques. This paper reports on the outcomes of an ontological analysis of BPMN and explores identified issues by reporting on interviews conducted with BPMN users in Australia. Complementing this analysis we consolidate our findings with previous ontological analyses of process modelling notations to deliver a comprehensive assessment of BPMN

Integrating form and meaning in L2 pronunciation instruction

One of the central challenges of ESL teaching is striking the right balance between form and meaning. In pronunciation pedagogy, this challenge is compounded because repetitive practice, which has been shown to enhance phonological acquisition and promote fluency, is widely viewed as being incompatible with communicative principles. This article provides a brief historical background for modern pronunciation pedagogy (from World War II to the present) as part of a backdrop for understanding the current disjuncture between pronunciation and communicative language teaching. A discussion on form-focused instruction, its applicability for pronunciation pedagogy, and challenges in implementation follows with reference to a recent article that presents evidence for the appropriateness of a communicative instructional framework for teaching L2 pronunciation (Trofimovich & Gatbonton, 2006). Finally, a communicative activity that encourages repetitive practice while integrating pronunciation with other components of language use is proposed

Adolescent Literacy and Textbooks: An Annotated Bibliography

A companion report to Carnegie's Time to Act, provides an annotated bibliography of research on textbook design and reading comprehension for fourth through twelfth grade, arranged by topic. Calls for a dialogue between publishers and researchers

Enhancing Decision Tree based Interpretation of Deep Neural Networks through L1-Orthogonal Regularization

One obstacle that so far prevents the introduction of machine learning models primarily in critical areas is the lack of explainability. In this work, a practicable approach of gaining explainability of deep artificial neural networks (NN) using an interpretable surrogate model based on decision trees is presented. Simply fitting a decision tree to a trained NN usually leads to unsatisfactory results in terms of accuracy and fidelity. Using L1-orthogonal regularization during training, however, preserves the accuracy of the NN, while it can be closely approximated by small decision trees. Tests with different data sets confirm that L1-orthogonal regularization yields models of lower complexity and at the same time higher fidelity compared to other regularizers.Comment: 8 pages, 18th IEEE International Conference on Machine Learning and Applications (ICMLA) 201