Adaptive text mining: Inferring structure from sequences

Text mining is about inferring structure from sequences representing natural language text, and may be defined as the process of analyzing text to extract information that is useful for particular purposes. Although hand-crafted heuristics are a common practical approach for extracting information from text, a general, and generalizable, approach requires adaptive techniques. This paper studies the way in which the adaptive techniques used in text compression can be applied to text mining. It develops several examples: extraction of hierarchical phrase structures from text, identification of keyphrases in documents, locating proper names and quantities of interest in a piece of text, text categorization, word segmentation, acronym extraction, and structure recognition. We conclude that compression forms a sound unifying principle that allows many text mining problems to be tacked adaptively

Mind: meet network. Emergence of features in conceptual metaphor.

As a human product, language reflects the psychological experience of man (Radden and Dirven, 2007). One model of language and human cognition in general is connectionism, by many linguists is regarded as mathematical and, therefore, too reductive. This opinion trend seems to be reversing, however, due to the fact that many cognitive researchers begin to appreciate one attribute of network models: feature emergence. In the course of a network simulation properties emerge that were neither inbuilt nor intended by its creators (Elman, 1998), in other words, the whole becomes more than just the sum of its parts. Insight is not only drawn from the network's output, but also the means that the network utilizes to arrive at the output.\ud It may seem obvious that the events of life should be meaningful for human beings, yet there is no widely accepted theory as to how do we derive that meaning. The most promising hypothesis regarding the question how the world is meaningful to us is that of embodied cognition (cf. Turner 2009), which postulates that the functions of the brain evolved so as to ‘understand’ the body, thus grounding the mind in an experiential foundation. Yet, the relationship between the body and the mind is far from perspicuous, as research insight is still intertwined with metaphors specific for the researcher’s methodology (Eliasmith 2003). It is the aim of this paper to investigate the conceptual metaphor in a manner that will provide some insight with regard to the role that objectification, as defined by Szwedek (2002), plays in human cognition and identify one possible consequence of embodied cognition.\ud If the mechanism for concept formation, or categorization of the world, resembles a network, it is reasonable to assume that evidence for this is to be sought in language. Let us then postulate the existence of a network mechanism for categorization and concept formation present in the human mind and initially developed to cope with the world directly accessible to the early human (i.e. tangible). Such a network would convert external inputs to form an internal, multi modal representation of a perceived object in the brain. The sheer amount of available information and the computational restrictions of the brain would force some sort of data compression, or a computational funnel. It has been shown that a visual perception network of this kind can learn to accurately label patterns (Elman, 1998). What is more, the compression of data facilitated the recognition of prototypes of a given pattern category rather than its peripheral representations, an emergent property that supports the prototype theory of the mental lexicon (cf. Radden and Dirven, 2007).\ud The present project proposes that, in the domain of cognition, the process of objectification, as defined by Szwedek (2002), would be an emergent property of such a system, or that if an abstract notion is computed by a neural network designed to cope with tangible concepts the data compression mechanism would require the notion to be conceptualized as an object to permit further processing. The notion of emergence of meaning from the operation of complex systems is recognised as an important process in a number of studies on metaphor comprehension. Feature emergence is said to occur when a non-salient feature of the target and the vehicle becomes highly salient in the metaphor (Utsumi 2005). Therefore, for example, should objectification emerge as a feature in the metaphor KNOWLEDGE IS A TREASURE, the metaphor would be characterised as having more\ud features of an object than either the target or vehicle alone. This paper focuses on providing a theoretical connectionist network based on the Elman-type network (Elman, 1998) as a model of concept formation where objectification would be an emergent feature. This is followed by a psychological experiment whereby the validity of this assumption is tested through a questionnaire where two groups of participants are asked to evaluate either metaphors or their components. The model proposes an underlying relation between the mechanism for concept formation and the omnipresence of conceptual metaphors, which are interpreted as resulting from the properties of the proposed network system.\ud Thus, an evolutionary neural mechanism is proposed for categorization of the world, that is able to cope with both concrete and abstract notions and the by-product of which are the abstract language-related phenomena, i.e. metaphors. The model presented in this paper aims at providing a unified account of how the various types of phenomena, objects, feelings etc. are categorized in the human mind, drawing on evidence from language.\ud References:\ud Szwedek, Aleksander. 2002. Objectification: From Object Perception To Metaphor Creation. In B. Lewandowska-Tomaszczyk and K. Turewicz (eds). Cognitive Linguistics To-day, 159-175. Frankfurt am Main: Peter Lang.\ud Radden, Günter and Dirven, René. 2007. Cognitive English Grammar. Amsterdam/ Philadelphia: John Benjamins Publishing Company\ud Eliasmith, Chris. 2003. Moving beyond metaphors: understanding the mind for what it is. Journal of Philosophy. C(10):493- 520.\ud Elman, J. L. et al. 1998. Rethinking innateness: A connectionist perspective on development. Cambridge, MA: MIT Press\ud Turner, Mark. 2009. Categorization of Time and Space Through Language. (Paper presented at the FOCUS2009 conference "Categorization of the world through language". Serock, 25-28 February 2009).\ud Utsumi, Akira. 2005. The role of feature emergence in metaphor appreciation, Metaphor and Symbol, 20(3), 151-172

Cross-lingual Distillation for Text Classification

Cross-lingual text classification(CLTC) is the task of classifying documents written in different languages into the same taxonomy of categories. This paper presents a novel approach to CLTC that builds on model distillation, which adapts and extends a framework originally proposed for model compression. Using soft probabilistic predictions for the documents in a label-rich language as the (induced) supervisory labels in a parallel corpus of documents, we train classifiers successfully for new languages in which labeled training data are not available. An adversarial feature adaptation technique is also applied during the model training to reduce distribution mismatch. We conducted experiments on two benchmark CLTC datasets, treating English as the source language and German, French, Japan and Chinese as the unlabeled target languages. The proposed approach had the advantageous or comparable performance of the other state-of-art methods.Comment: Accepted at ACL 2017; Code available at https://github.com/xrc10/cross-distil

Towards Effective Codebookless Model for Image Classification

The bag-of-features (BoF) model for image classification has been thoroughly studied over the last decade. Different from the widely used BoF methods which modeled images with a pre-trained codebook, the alternative codebook free image modeling method, which we call Codebookless Model (CLM), attracted little attention. In this paper, we present an effective CLM that represents an image with a single Gaussian for classification. By embedding Gaussian manifold into a vector space, we show that the simple incorporation of our CLM into a linear classifier achieves very competitive accuracy compared with state-of-the-art BoF methods (e.g., Fisher Vector). Since our CLM lies in a high dimensional Riemannian manifold, we further propose a joint learning method of low-rank transformation with support vector machine (SVM) classifier on the Gaussian manifold, in order to reduce computational and storage cost. To study and alleviate the side effect of background clutter on our CLM, we also present a simple yet effective partial background removal method based on saliency detection. Experiments are extensively conducted on eight widely used databases to demonstrate the effectiveness and efficiency of our CLM method

A Taxonomy of Big Data for Optimal Predictive Machine Learning and Data Mining

Big data comes in various ways, types, shapes, forms and sizes. Indeed, almost all areas of science, technology, medicine, public health, economics, business, linguistics and social science are bombarded by ever increasing flows of data begging to analyzed efficiently and effectively. In this paper, we propose a rough idea of a possible taxonomy of big data, along with some of the most commonly used tools for handling each particular category of bigness. The dimensionality p of the input space and the sample size n are usually the main ingredients in the characterization of data bigness. The specific statistical machine learning technique used to handle a particular big data set will depend on which category it falls in within the bigness taxonomy. Large p small n data sets for instance require a different set of tools from the large n small p variety. Among other tools, we discuss Preprocessing, Standardization, Imputation, Projection, Regularization, Penalization, Compression, Reduction, Selection, Kernelization, Hybridization, Parallelization, Aggregation, Randomization, Replication, Sequentialization. Indeed, it is important to emphasize right away that the so-called no free lunch theorem applies here, in the sense that there is no universally superior method that outperforms all other methods on all categories of bigness. It is also important to stress the fact that simplicity in the sense of Ockham's razor non plurality principle of parsimony tends to reign supreme when it comes to massive data. We conclude with a comparison of the predictive performance of some of the most commonly used methods on a few data sets.Comment: 18 pages, 2 figures 3 table