Aprendizaje activo para clasificación de preguntas

Quepy es una librería para construir sistema de respuesta a preguntas sobre datos enlazados, sin embargo utiliza patrones estáticos para reconocer preguntas y alcanzar una gran cobertura es muy costoso. Utilizamos un clasificador bayesiano ingenuo para clasificar reformulaciones de preguntas semánticamente equivalentes, ligarlas a una misma interpretación y aumentar esta cobertura. La falta de un corpus de preguntas etiquetado nos llevó a utilizar aprendizaje activo tanto sobre instancias como sobre características. En la representación de las instancias incluimos también las concordancias con los patrones parciales del Quepy, para capturar la información semántica de la pregunta. En este escenario contamos con muchas de preguntas que no concuerdan con ningún patrón y muchas clases minoritarias con pocos ejemplos cada una. Los resultados indican que balancear el corpus de entrenamiento utilizado por el clasificador evita que la clase mayoritaria se convierta en la única clase reconocida, mientras que el entrenamiento con características aumentó en gran medida el reconocimiento de las clases minoritarias

Estudio de representaciones mediante co-embeddings para estudiantes y contenidos en minerı́a de datos educativos

Tesis (Doctora en Ciencias de la Computación)--Universidad Nacional de Córdoba, Facultad de Matemática, Astronomía, Física y Computación, 2019.Este trabajo es un estudio sobre la generación automática de representaciones basadas en métodos neuronales, en aplicaciones dentro del área de Minerı́a de Datos Educacionales (EDM). Se propone utilizar una arquitectura neuronal recurrente para modelar el cambio en el estado de los estudiantes a medida que interactúan con plataformas de aprendizaje en lı́nea. Al mismo tiempo, se generan representaciones automáticas para los elementos de los cursos, como problemas o lecciones, evitando la necesidad de utilizar ejemplos etiquetados con información adicional, y en consecuencia costosos de obtener. Sobre esta base, se modifica la arquitectura para modelar explı́citamente la relación entre la representación de los estudiantes y la de los componentes del curso, proyectando ambos tipos de entidades en el mismo espacio latente. De esta manera, se espera mejorar el desempeño del clasificador a través de la inyección directa de conocimiento de dominio en el modelo. Ambas propuestas son evaluadas para las tareas de estimación de conocimiento (Knowledge Tracing) y predicción del abandono escolar (dropout) en tutores inteligentes y cursos masivos, respectivamente. Se observa que las representaciones conjuntas de estudiantes y lecciones obtienen resultados similares a las representaciones disjuntas, mejorando significativamente en escenarios con pocos datos o con desbalance de clases pronunciado.This work is a study on the automatic generation of representations based on neuronal methods, for applications in the area of Educational Data Mining (EDM). We proposed to use a recurrent neuronal architecture to model the change in the state of students as they interact with online learning platforms. At the same time, automatic representations are generated for course elements, such as problems or lessons, avoiding the need to use examples labeled with additional information, and consequently costly to obtain. On this basis, the architecture is modified to explicitly model the relationship between the students’ representation and that of the course components, projecting both types of entities in the same latent space. In this way, the performance of the classifier is expected to improve through the direct injection of domain knowledge into the model. Both proposals are evaluated for knowledge tracing and dropout prediction in intelligent tutor systems and mass open online courses, respectively. It is observed that the joint representations of students and lessons obtain results similar to the disjoint representations, improving significantly in scenarios with fewer training data, partial sequences, or with pronounced class imbalance.Fil: Teruel, Milagro. Universidad Nacional de Córdoba. Facultad de Matemática, Astronomía, Física y Computación; Argentina

Reversing uncertainty sampling to improve active learning schemes

Ponencia presentada en el 16º Simposio Argentino de Inteligencia Artificial. 44 Jornadas Argentinas de Informática. Rosario, Argentina, del 31 de agosto al 4 de septiembre de 2015.Active learning provides promising methods to optimize the cost of manually annotating a dataset. However, practitioners in many areas do not massively resort to such methods because they present technical difficulties and do not provide a guarantee of good performance, especially in skewed distributions with scarcely populated minority classes and an undefined, catch-all majority class, which are very common in human-related phenomena like natural language. In this paper we present a comparison of the simplest active learning technique, pool-based uncertainty sampling, and its opposite, which we call reversed uncertainty sampling. We show that both obtain results comparable to the random, arguing for a more insightful approach to active learning.http://44jaiio.sadio.org.ar/asaiFil: Cardellino, Cristian Adrián. Universidad Nacional de Córdoba. Facultad de Matemática, Astronomía y Física; Argentina.Fil: Teruel, Milagro. Universidad Nacional de Córdoba. Facultad de Matemática, Astronomía y Física; Argentina.Fil: Alonso i Alemany, Laura. Universidad Nacional de Córdoba. Facultad de Matemática, Astronomía y Física; Argentina.Ciencias de la Computació

Different Flavors of Attention Networks for Argument Mining

International audienceArgument mining is a rising area of Natural Language Processing (NLP) concerned with the automatic recognition and interpretation of argument components and their relations. Neural models are by now mature technologies to be exploited for automating the argument mining tasks, despite the issue of data sparseness. This could ease much of the manual effort involved in these tasks, taking into account heterogeneous types of texts and topics. In this work, we evaluate different attention mechanisms applied over a state-of-the-art architecture for sequence labeling. We assess the impact of different flavors of attention in the task of argument component detection over two datasets: essays and legal domain. We show that attention not only models the problem better but also supports interpretability

Combining semi-supervised and active learning to recognize minority senses in a new corpus

Ponencia presentada en la 24th International Joint Conference on Artificial Intelligence. Workshop on Replicability and Reproducibility in Natural Language Processing: adaptive methods, resources and software. Buenos Aires, Argentina, del 25 al 31 de julio de 2015.In this paper we study the impact of combining active learning with bootstrapping to grow a small annotated corpus from a different, unannotated corpus. The intuition underlying our approach is that bootstrapping includes instances that are closer to the generative centers of the data, while discriminative approaches to active learning include instances that are closer to the decision boundaries of classifiers. We build an initial model from the original annotated corpus, which is then iteratively enlarged by including both manually annotated examples and automatically labelled examples as training examples for the following iteration. Examples to be annotated are selected in each iteration by applying active learning techniques. We show that intertwining an active learning component in a bootstrapping approach helps to overcome an initial bias towards a majority class, thus facilitating adaptation of a starting dataset towards the real distribution of a different, unannotated corpus.Fil: Cardellino, Cristian Adrián. Universidad Nacional de Córdoba. Facultad de Matemática, Astronomía y Física; Argentina.Fil: Teruel, Milagro. Universidad Nacional de Córdoba. Facultad de Matemática, Astronomía y Física; Argentina.Fil: Alonso i Alemany, Laura. Universidad Nacional de Córdoba. Facultad de Matemática, Astronomía y Física; Argentina.Otras Ciencias de la Computación e Informació

On Improving Summarization Factual Consistency from Natural Language Feedback

Despite the recent progress in language generation models, their outputs may not always meet user expectations. In this work, we study whether informational feedback in natural language can be leveraged to improve generation quality and user preference alignment. To this end, we consider factual consistency in summarization, the quality that the summary should only contain information supported by the input documents, as the user-expected preference. We collect a high-quality dataset, DeFacto, containing human demonstrations and informational natural language feedback consisting of corrective instructions, edited summaries, and explanations with respect to the factual consistency of the summary. Using our dataset, we study three natural language generation tasks: (1) editing a summary by following the human feedback, (2) generating human feedback for editing the original summary, and (3) revising the initial summary to correct factual errors by generating both the human feedback and edited summary. We show that DeFacto can provide factually consistent human-edited summaries and further insights into summarization factual consistency thanks to its informational natural language feedback. We further demonstrate that fine-tuned language models can leverage our dataset to improve the summary factual consistency, while large language models lack the zero-shot learning ability in our proposed tasks that require controllable text generation.Comment: ACL 2023 Camera Ready, GitHub Repo: https://github.com/microsoft/DeFact

Learning Slowly To Learn Better: Curriculum Learning for Legal Ontology Population

International audienceIn this paper, we present an ontology population approach for legal ontologies. We exploit Wikipedia as a source of manually annotated examples of legal entities. We align YAGO, a Wikipedia-based ontology, and LKIF, an ontology specifically designed for the legal domain. Through this alignment, we can effectively populate the LKIF ontology, with the aim to obtain examples to train a Named Entity Recognizer and Classifier to be used for finding and classifying entities in legal texts. Since examples of annotated data in the legal domain are very few, we apply a machine learning strategy called curriculum learning aimed to overcome problems of overfitting by learning increasingly more complex concepts. We compare the performance of this method to identify Named Entities with respect to batch learning as well as two other baselines. Results are satisfying and foster further research in this direction

Legal NERC with ontologies, Wikipedia and curriculum learning

International audienceIn this paper, we present a Wikipedia-based approach to develop resources for the legal domain. We establish a mapping between a legal domain ontology, LKIF (Hoekstra et al., 2007), and a Wikipedia-based ontology, YAGO (Suchanek et al., 2007), and through that we populate LKIF. Moreover, we use the mentions of those entities in Wikipedia text to train a specific Named Entity Recognizer and Classifier. We find that this classifier works well in the Wikipedia, but, as could be expected, performance decreases in a corpus of judgments of the European Court of Human Rights. However, this tool will be used as a preprocess for human annotation. We resort to a technique called curriculum learning aimed to overcome problems of overfitting by learning increasingly more complex concepts. However, we find that in this particular setting, the method works best by learning from most specific to most general concepts, not the other way round

Improving Grounded Language Understanding in a Collaborative Environment by Interacting with Agents Through Help Feedback

Many approaches to Natural Language Processing (NLP) tasks often treat them as single-step problems, where an agent receives an instruction, executes it, and is evaluated based on the final outcome. However, human language is inherently interactive, as evidenced by the back-and-forth nature of human conversations. In light of this, we posit that human-AI collaboration should also be interactive, with humans monitoring the work of AI agents and providing feedback that the agent can understand and utilize. Further, the AI agent should be able to detect when it needs additional information and proactively ask for help. Enabling this scenario would lead to more natural, efficient, and engaging human-AI collaborations. In this work, we explore these directions using the challenging task defined by the IGLU competition, an interactive grounded language understanding task in a MineCraft-like world. We explore multiple types of help players can give to the AI to guide it and analyze the impact of this help in AI behavior, resulting in performance improvements

Different Flavors of Attention Networks for Argument Mining

International audienceArgument mining is a rising area of Natural Language Processing (NLP) concerned with the automatic recognition and interpretation of argument components and their relations. Neural models are by now mature technologies to be exploited for automating the argument mining tasks, despite the issue of data sparseness. This could ease much of the manual effort involved in these tasks, taking into account heterogeneous types of texts and topics. In this work, we evaluate different attention mechanisms applied over a state-of-the-art architecture for sequence labeling. We assess the impact of different flavors of attention in the task of argument component detection over two datasets: essays and legal domain. We show that attention not only models the problem better but also supports interpretability