Forgetting Exceptions is Harmful in Language Learning

We show that in language learning, contrary to received wisdom, keeping exceptional training instances in memory can be beneficial for generalization accuracy. We investigate this phenomenon empirically on a selection of benchmark natural language processing tasks: grapheme-to-phoneme conversion, part-of-speech tagging, prepositional-phrase attachment, and base noun phrase chunking. In a first series of experiments we combine memory-based learning with training set editing techniques, in which instances are edited based on their typicality and class prediction strength. Results show that editing exceptional instances (with low typicality or low class prediction strength) tends to harm generalization accuracy. In a second series of experiments we compare memory-based learning and decision-tree learning methods on the same selection of tasks, and find that decision-tree learning often performs worse than memory-based learning. Moreover, the decrease in performance can be linked to the degree of abstraction from exceptions (i.e., pruning or eagerness). We provide explanations for both results in terms of the properties of the natural language processing tasks and the learning algorithms.Comment: 31 pages, 7 figures, 10 tables. uses 11pt, fullname, a4wide tex styles. Pre-print version of article to appear in Machine Learning 11:1-3, Special Issue on Natural Language Learning. Figures on page 22 slightly compressed to avoid page overloa

Cognitive finance: Behavioural strategies of spending, saving, and investing.

Research in economics is increasingly open to empirical results. The advances in behavioural approaches are expanded here by applying cognitive methods to financial questions. The field of "cognitive finance" is approached by the exploration of decision strategies in the financial settings of spending, saving, and investing. Individual strategies in these different domains are searched for and elaborated to derive explanations for observed irregularities in financial decision making. Strong context-dependency and adaptive learning form the basis for this cognition-based approach to finance. Experiments, ratings, and real world data analysis are carried out in specific financial settings, combining different research methods to improve the understanding of natural financial behaviour. People use various strategies in the domains of spending, saving, and investing. Specific spending profiles can be elaborated for a better understanding of individual spending differences. It was found that people differ along four dimensions of spending, which can be labelled: General Leisure, Regular Maintenance, Risk Orientation, and Future Orientation. Saving behaviour is strongly dependent on how people mentally structure their finance and on their self-control attitude towards decision space restrictions, environmental cues, and contingency structures. Investment strategies depend on how companies, in which investments are placed, are evaluated on factors such as Honesty, Prestige, Innovation, and Power. Further on, different information integration strategies can be learned in decision situations with direct feedback. The mapping of cognitive processes in financial decision making is discussed and adaptive learning mechanisms are proposed for the observed behavioural differences. The construal of a "financial personality" is proposed in accordance with other dimensions of personality measures, to better acknowledge and predict variations in financial behaviour. This perspective enriches economic theories and provides a useful ground for improving individual financial services

Towards Efficient Lifelong Machine Learning in Deep Neural Networks

Humans continually learn and adapt to new knowledge and environments throughout their lifetimes. Rarely does learning new information cause humans to catastrophically forget previous knowledge. While deep neural networks (DNNs) now rival human performance on several supervised machine perception tasks, when updated on changing data distributions, they catastrophically forget previous knowledge. Enabling DNNs to learn new information over time opens the door for new applications such as self-driving cars that adapt to seasonal changes or smartphones that adapt to changing user preferences. In this dissertation, we propose new methods and experimental paradigms for efficiently training continual DNNs without forgetting. We then apply these methods to several visual and multi-modal perception tasks including image classification, visual question answering, analogical reasoning, and attribute and relationship prediction in visual scenes

A dual process account of creative thinking

This article explicates the potential role played by type 1 thinking (automatic, fast) and type 2 thinking (effortful, logical) in creative thinking. The relevance of Evans's (2007) models of conflict of dual processes in thinking is discussed with regards to creative thinking. The role played by type 1 thinking and type 2 thinking during the different stages of creativity (problem finding and conceptualization, incubation, illumination, verification and dissemination) is discussed. It is proposed that although both types of thinking are active in creativity, the extent to which they are active and the nature of their contribution to creativity will vary between stages of the creative process. Directions for future research to test this proposal are outlined; differing methodologies and the investigation of different stages of creative thinking are discussed. © Taylor & Francis Group, LLC

Rationale-Enhanced Language Models are Better Continual Relation Learners

Continual relation extraction (CRE) aims to solve the problem of catastrophic forgetting when learning a sequence of newly emerging relations. Recent CRE studies have found that catastrophic forgetting arises from the model's lack of robustness against future analogous relations. To address the issue, we introduce rationale, i.e., the explanations of relation classification results generated by large language models (LLM), into CRE task. Specifically, we design the multi-task rationale tuning strategy to help the model learn current relations robustly. We also conduct contrastive rationale replay to further distinguish analogous relations. Experimental results on two standard benchmarks demonstrate that our method outperforms the state-of-the-art CRE models.Comment: Accepted at EMNLP 202

From chunks to function-argument structure : a similarity-based approach

Chunk parsing has focused on the recognition of partial constituent structures at the level of individual chunks. Little attention has been paid to the question of how such partial analyses can be combined into larger structures for complete utterances. Such larger structures are not only desirable for a deeper syntactic analysis. They also constitute a necessary prerequisite for assigning function-argument structure. The present paper offers a similaritybased algorithm for assigning functional labels such as subject, object, head, complement, etc. to complete syntactic structures on the basis of prechunked input. The evaluation of the algorithm has concentrated on measuring the quality of functional labels. It was performed on a German and an English treebank using two different annotation schemes at the level of function argument structure. The results of 89.73% correct functional labels for German and 90.40%for English validate the general approach