End-user interactions with intelligent and autonomous systems.

Systems that learn from or personalize themselves to users are quickly becoming mainstream yet interaction with these systems is limited and often uninformative for the end user. This workshop focuses on approaches and challenges to explore making these systems transparent, controllable and ultimately trustworthy to end users. The aims of the workshop are to help establish connections among researchers and industrial practitioners using real-world problems as catalysts to facilitate the exchange of approaches, solutions, and ideas about how to better support end users

Explanatory debugging: Supporting end-user debugging of machine-learned programs

Many machine-learning algorithms learn rules of behavior from individual end users, such as task-oriented desktop organizers and handwriting recognizers. These rules form a “program” that tells the computer what to do when future inputs arrive. Little research has explored how an end user can debug these programs when they make mistakes. We present our progress toward enabling end users to debug these learned programs via a Natural Programming methodology. We began with a formative study exploring how users reason about and correct a text-classification program. From the results, we derived and prototyped a concept based on “explanatory debugging”, then empirically evaluated it. Our results contribute methods for exposing a learned program's logic to end users and for eliciting user corrections to improve the program's predictions

Towards recognizing "cool": can end users help computer vision recognize subjective attributes of objects in images?

Recent computer vision approaches are aimed at richer image interpretations that extend the standard recognition of objects in images (e.g., cars) to also recognize object attributes (e.g., cylindrical, has-stripes, wet). However, the more idiosyncratic and abstract the notion of an object attribute (e.g., cool car), the more challenging the task of attribute recognition. This paper considers whether end users can help vision algorithms recognize highly idiosyncratic attributes, referred to here as subjective attributes. We empirically investigated how end users recognized three subjective attributes of carscool, cute, and classic. Our results suggest the feasibility of vision algorithms recognizing subjective attributes of objects, but an interactive approach beyond standard supervised learning from labeled training examples is needed

Interacting meaningfully with machine learning systems: Three experiments

Although machine learning is becoming commonly used in today's software, there has been little research into how end users might interact with machine learning systems, beyond communicating simple “right/wrong” judgments. If the users themselves could work hand-in-hand with machine learning systems, the users’ understanding and trust of the system could improve and the accuracy of learning systems could be improved as well. We conducted three experiments to understand the potential for rich interactions between users and machine learning systems. The first experiment was a think-aloud study that investigated users’ willingness to interact with machine learning reasoning, and what kinds of feedback users might give to machine learning systems. We then investigated the viability of introducing such feedback into machine learning systems, specifically, how to incorporate some of these types of user feedback into machine learning systems, and what their impact was on the accuracy of the system. Taken together, the results of our experiments show that supporting rich interactions between users and machine learning systems is feasible for both user and machine. This shows the potential of rich human–computer collaboration via on-the-spot interactions as a promising direction for machine learning systems and users to collaboratively share intelligence

End-user feature engineering in the presence of class imbalance

Intelligent user interfaces, such as recommender systems and email classifiers, use machine learning algorithms to customize their behavior to the preferences of an end user. Although these learning systems are somewhat reliable, they are not perfectly accurate. Traditionally, end users who need to correct these learning systems can only provide more labeled training data. In this paper, we focus on incorporating new features suggested by the end user into machine learning systems. To investigate the effects of user-generated features on accuracy we developed an auto- coding application that enables end users to assist a machine-learned program in coding a transcript by adding custom features. Our results show that adding user-generated features to the machine learning algorithm can result in modest improvements to its F1 score. Further improvements are possible if the algorithm accounts for class imbalance in the training data and deals with low-quality user-generated features that add noise to the learning algorithm. We show that addressing class imbalance improves performance to an extent but improving the quality of features brings about the most beneficial change. Finally, we discuss changes to the user interface that can help end users avoid the creation of low-quality features.Keywords: Feature Engineering, Class Imbalance, machine learning, artificial intelligence, end-user programming, HC

Interacting meaningfully with machine learning systems : three experiments

Although machine learning is becoming commonly used in today's software, there has been little research into how end users might interact with machine learning systems, beyond communicating simple "right/wrong" judgments. If the users themselves could somehow work hand-in-hand with machine learning systems, the accuracy of learning systems could be improved and the users' understanding and trust of the system could improve as well. We conducted three experiments to begin to understand the potential for rich interactions between users and machine learning systems. The first experiment was a think-aloud study, aiming to see how willing users were to interact with and about machine learning reasoning, and to help us understand what kinds of feedback users might give to machine learning systems. Specifically, users were shown explanations of machine learning predictions and asked to provide feedback to improve the predictions. The results were that users' feedback was rich, complex, and widely varied, ranging from suggestions for reweighting of features to proposals for new features, feature combinations, relational features, and wholesale changes to the learning algorithm. We then investigated the viability of introducing such feedback into machine learning systems: specifically, how to incorporate some of these types of user feedback into machine learning systems, and impact on the accuracy of the system. Taken together, the results of our experiments show that supporting rich interactions between users and machine learning systems is feasible for both user and machine. This shows the potential of rich human-computer collaboration via on-the-spot interactions as a promising direction for machine learning systems to work more intelligently, hand-in-hand with the user

Abordagem modular baseada em dicionário para reconhecimento de entidades nomeadas através de associação aproximada

Orientador : Marcos Didonet Del FabroDissertação (mestrado) - Universidade Federal do Paraná, Setor de Ciências Exatas, Programa de Pós-Graduação em Informática. Defesa: Curitiba, 29/08/2016Inclui referências : f. 47-50Área de concentração: Ciência da computaçãoResumo: As técnicas de extração de informações estão sempre evoluindo para serem capazes de trabalhar com a quantidade crescente de dados disponíveis através de textos em linguagem natural e não estruturados. Destacamos a subtarefa da extração de informação conhecida como reconhecimento de entidades nomeadas baseado em dicionário, que realiza a identificação de sequências de caracteres que representam entidades de um determinado grupo, e o bom desempenho dessa subtarefa é fundamental para um bom processo de extração de informação. O reconhecimento de entidades nomeadas (NER) permite definir os sujeitos que são abordados pelo texto como organizações, pessoas, locais, etc. Pontos que ainda são desafios dentro da subtarefa de NER para sistemas baseados em dicionário são a presença de erros ortográficos nos textos e a existência de poucos sistemas de NER capazes de trabalhar em diferentes contextos. Esse trabalho apresenta uma abordagem para o reconhecimento de entidades nomeadas baseado em dicionário. Para trabalhar com textos que podem apresentar erros ortográficos, é utilizada uma busca por associação aproximada baseada na distância de edição entre as sequências de caracteres que representam a entrada do dicionário e as sub-partes do texto. Para promover a redução do erro entre as sequências de caracteres (SC) e facilitar a busca por associação aproximada são utilizados algoritmos de transformação. Esses algoritmos permitem a busca sobre o dicionário encontrar uma quantidade maior de entidades se comparada com as buscas utilizando as SCs originais para um mesmo valor da distância de edição aceita. As transformações também colaboram com a redução do tamanho das SCs e com a criação de mais prefixos similares, promovendo uma redução no tamanho da árvore de prefixo que indexa o dicionário. Para melhorar a precisão da nossa abordagem, disponibilizamos recursos de filtragem que fazem uso de métricas de similaridade para eliminar entidades falsas que foram retornadas da busca sobre o dicionário. Nossa abordagem também foi projetada para permitir a configuração de alguns componentes de forma a ser adaptada para diferentes casos de estudo. Palavras-chave: Reconhecimento de entidades nomeadas, Associação Aproximada de Sequências de Caracteres, Conversão fonética.Abstract: The information extraction techniques are always evolving to be able to work with the increasing amount of unstructured data available through texts in natural language. We highlight the information extraction subtask known as dictionary-based named entity recognition, which performs the identification of strings that represent entities of a particular group, and the good performance of this sub-task is critical for a good extracting information process. The named entity recognition (NER) defines the nouns that are covered by the text as organizations, people, places, etc. Some subjects that still represent chalenges in the sub-task of NER for currently systems that are dictionary-based are the presence of spelling errors in the text and the existence of few NER systems that are able to work in different contexts. This work presents an approach of a dictionary-based named entity recognition. Looking to work with texts that may have spelling errors, we use an approximate string matching search based on edit distance between the strings that represent the entries of the dictionary and the substrings of the text. To further the reduction of the error between the strings and facilitate the search using approximate matching we used transformation algorithms. These algorithms allow the search on the dictionary find a greater amount of entities if compared with the search using the original strings, for the same value of Edit Distance. Transformations also promote the strings size reduction and create more similar prefixes, promoting a reduction in the size of the prefix tree (trie) that indexes the dictionary. To improve the precision of our approach, we provide filtering capabilities that make use of similarity metrics to eliminate false entities that have been returned from the search on the dictionary trie. Our approach is also designed to enable the configuration of some components to be adapted to different study cases. Keywords: Named entity recognition, Approximate string matching, Phonetic conversion

Diverse Contributions to Implicit Human-Computer Interaction

Cuando las personas interactúan con los ordenadores, hay mucha información que no se proporciona a propósito. Mediante el estudio de estas interacciones implícitas es posible entender qué características de la interfaz de usuario son beneficiosas (o no), derivando así en implicaciones para el diseño de futuros sistemas interactivos. La principal ventaja de aprovechar datos implícitos del usuario en aplicaciones informáticas es que cualquier interacción con el sistema puede contribuir a mejorar su utilidad. Además, dichos datos eliminan el coste de tener que interrumpir al usuario para que envíe información explícitamente sobre un tema que en principio no tiene por qué guardar relación con la intención de utilizar el sistema. Por el contrario, en ocasiones las interacciones implícitas no proporcionan datos claros y concretos. Por ello, hay que prestar especial atención a la manera de gestionar esta fuente de información. El propósito de esta investigación es doble: 1) aplicar una nueva visión tanto al diseño como al desarrollo de aplicaciones que puedan reaccionar consecuentemente a las interacciones implícitas del usuario, y 2) proporcionar una serie de metodologías para la evaluación de dichos sistemas interactivos. Cinco escenarios sirven para ilustrar la viabilidad y la adecuación del marco de trabajo de la tesis. Resultados empíricos con usuarios reales demuestran que aprovechar la interacción implícita es un medio tanto adecuado como conveniente para mejorar de múltiples maneras los sistemas interactivos.Leiva Torres, LA. (2012). Diverse Contributions to Implicit Human-Computer Interaction [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/17803Palanci