Identifying beneficial task relations for multi-task learning in deep neural networks

Multi-task learning (MTL) in deep neural networks for NLP has recently received increasing interest due to some compelling benefits, including its potential to efficiently regularize models and to reduce the need for labeled data. While it has brought significant improvements in a number of NLP tasks, mixed results have been reported, and little is known about the conditions under which MTL leads to gains in NLP. This paper sheds light on the specific task relations that can lead to gains from MTL models over single-task setups.Comment: Accepted for publication at EACL 201

Transferable Output ASCII Data (TOAD) gateway: Version 1.0 user's guide

The Transferable Output ASCII Data (TOAD) Gateway, release 1.0 is described. This is a software tool for converting tabular data from one format into another via the TOAD format. This initial release of the Gateway allows free data interchange among the following file formats: TOAD; Standard Interface File (SIF); Program to Optimize Simulated Trajectories (POST) input; Comma Separated Value (TSV); and a general free-form file format. As required, additional formats can be accommodated quickly and easily

Transferable Output ASCII Data (TOAD) file format description

Described is a format for writing ASCII data on a file to facilitate its transfer from one computer system to another. The TOAD format conforms to all ANSI FORTRAN 77 standards. There are two advantages in using the TOAD format. First, TOAD files are of the preferred type and record length to make them easy to edit, read from and write on magnetic tape, or transfer across communications networks. Secondly, application programs, using the TOAD format to write computational results, are more portable and the answer files easier to postprocess. TOAD utility software is listed in an appendix

Latent Multi-task Architecture Learning

Multi-task learning (MTL) allows deep neural networks to learn from related tasks by sharing parameters with other networks. In practice, however, MTL involves searching an enormous space of possible parameter sharing architectures to find (a) the layers or subspaces that benefit from sharing, (b) the appropriate amount of sharing, and (c) the appropriate relative weights of the different task losses. Recent work has addressed each of the above problems in isolation. In this work we present an approach that learns a latent multi-task architecture that jointly addresses (a)--(c). We present experiments on synthetic data and data from OntoNotes 5.0, including four different tasks and seven different domains. Our extension consistently outperforms previous approaches to learning latent architectures for multi-task problems and achieves up to 15% average error reductions over common approaches to MTL.Comment: To appear in Proceedings of AAAI 201

Named entity tagging a very large unbalanced corpus: training and evaluating NE classifiers

We describe a systematic and application-oriented approach to training and evaluating named entity recognition and classification (NERC) systems, the purpose of which is to identify an optimal system and to train an optimal model for named entity tagging DeReKo, a very large general-purpose corpus of contemporary German (Kupietz et al., 2010). DeReKo 's strong dispersion wrt. genre, register and time forces us to base our decision for a specific NERC system on an evaluation performed on a representative sample of DeReKo instead of performance figures that have been reported for the individual NERC systems when evaluated on more uniform and less diverse data. We create and manually annotate such a representative sample as evaluation data for three different NERC systems, for each of which various models are learnt on multiple training data. The proposed sampling method can be viewed as a generally applicable method for sampling evaluation data from an unbalanced target corpus for any sort of natural language processing

Analytic Patch Configuration (APC) gateway version 1.0 user's guide

The Analytic Patch Configuration (APC) is an interactive software tool which translates aircraft configuration geometry files from one format into another. This initial release of the APC Gateway accommodates six formats: the four accepted APC formats (89f, 89fd, 89u, and 89ud), the PATRAN 2.x phase 1 neutral file format, and the Integrated Aerodynamic Analysis System (IAAS) General Geometry (GG) format. Written in ANSI FORTRAN 77 and completely self-contained, the APC Gateway is very portable and was already installed on CDC/NOS, VAX/VMS, SUN, SGI/IRIS, CONVEX, and GRAY hosts

Exchange and correlation near the nucleus in density functional theory

The near nucleus behavior of the exchange-correlation potential $v_{xc}({\bf r})$ in Hohenberg-Kohn-Sham density functional theory is investigated. It is shown that near the nucleus the linear term of $O(r)$ of the spherically averaged exchange-correlation potential ${\bar v}_{xc}(r)$ is nonzero, and that it arises purely from the difference between the kinetic energy density at the nucleus of the interacting system and the noninteracting Kohn-Sham system. An analytical expression for the linear term is derived. Similar results for the exchange $v_{x}({\bf r})$ and correlation $v_{c}({\bf r})$ potentials are also obtained separately. It is further pointed out that the linear term in $v_{xc}({\bf r})$ arising mainly from $v_{c}({\bf r})$ is rather small, and $v_{xc}({\bf r})$ therefore has a nearly quadratic structure near the nucleus. Implications of the results for the construction of the Kohn-Sham system are discussed with examples.Comment: 10 page

Disembodied Machine Learning: On the Illusion of Objectivity in NLP

Machine Learning seeks to identify and encode bodies of knowledge within provided datasets. However, data encodes subjective content, which determines the possible outcomes of the models trained on it. Because such subjectivity enables marginalisation of parts of society, it is termed (social) `bias' and sought to be removed. In this paper, we contextualise this discourse of bias in the ML community against the subjective choices in the development process. Through a consideration of how choices in data and model development construct subjectivity, or biases that are represented in a model, we argue that addressing and mitigating biases is near-impossible. This is because both data and ML models are objects for which meaning is made in each step of the development pipeline, from data selection over annotation to model training and analysis. Accordingly, we find the prevalent discourse of bias limiting in its ability to address social marginalisation. We recommend to be conscientious of this, and to accept that de-biasing methods only correct for a fraction of biases.Comment: In revie