Few-Shot Natural Language Processing by Meta-Learning Without Labeled Data

Humans show a remarkable capability to accurately solve a wide range of problems efficiently -- utilizing a limited amount of computation and experience. Deep learning models, by stark contrast, can be trained to be highly accurate on a narrow task while being highly inefficient in terms of the amount of compute and data required to reach that accuracy. Within natural language processing (NLP), recent breakthroughs in unsupervised pretraining have enabled reusable models that can be applied to many NLP tasks, however, learning of new tasks is still inefficient. This has led to research on few-shot learning, where the goal is to generalize to new tasks with very few labeled instances. Meta-learning, or learning to learn, treats the learning process itself as a learning problem from data with the goal of learning systems that can generalize to new tasks efficiently. This has the potential to produce few-shot learners that can accurately solve a wide range of new tasks. However, meta-learning requires a distribution over tasks with relevant labeled data that can be difficult to obtain, severely limiting the practical utility of meta-learning methods. In this dissertation, we develop methods to enable large-scale meta-learning from unlabeled text data and improve the few-shot generalization ability of NLP models. We contribute methods that propose tasks synthetically created from unlabeled text, allowing for a large task distribution for meta-learning. This leads to rapid learning of new tasks by meta-learning from millions of self-supervised tasks and minimizes the train-test mismatch in few-shot learning by optimizing the pre-training directly for future fine-tuning with a few examples. Since real-world applications of NLP require learning diverse tasks with different numbers of classes, we first introduce an optimization-based meta-learning method that can learn from multiple NLP classification tasks with any number of classes. We then leverage the proposed self-supervised approach to create meta-training tasks, with a diverse number of classes, and meta-train models for few-shot learning using this task distribution. This leads to better representation learning, learning key hyper-parameters like learning rates, can be combined with supervised tasks to regularize supervised meta-learning, and leads to accurate few-shot learning on a diverse set of NLP classification tasks. We further explore the space of self-supervised tasks for meta-learning by considering important aspects like task diversity, difficulty, type, domain, and curriculum, and investigate how they affect meta-learning performance. Our analysis shows that all these factors meaningfully alter the task distribution, some inducing significant improvements in downstream few-shot accuracy of the meta-learned models. Our findings yield accurate and efficient meta-learning methods that improve few-shot generalization to diverse tasks and should enable many future applications of meta-learning in NLP, such as hyper-parameter optimization, continual learning, efficient learning, learning in low-resource languages, and more

Dynamic Treatment Regimes with Interference

Precision medicine describes healthcare in which patient-level data are used to inform treatment decisions. Within this framework, dynamic treatment regimes (DTRs) are sequences of decision rules that take individual patient information as input, and then output treatment recommendations. The primary purpose of DTR research is to estimate the optimal dynamic treatment regimes: the sequence of treatment rules that will optimize some pre-defined outcomes across a population. The focus of this thesis is on developing methods for estimating optimal DTRs in the presence of interference, where one patient’s outcome can be affected by others’ treatment. DTR estimation methods typically rely on the assumption of no interference. In many social network contexts, such as friendship or family networks, and for many health concerns, such as infectious diseases, this assumption is questionable. Moreover, the existing doubly robust regression-based DTR estimation methods are primarily focused on continuous outcomes. DTR estimation methods for binary or ordinal outcomes are more complicated due to less information being provided by these discrete outcomes. Consequently, very few DTR estimation methods focus on binary or ordinal outcomes, let alone methods when interference is present. To address these problems, for continuous outcomes, we directly establish novel interference-aware DTR estimation methods, and for binary or ordinal outcomes, we develop methods for DTR estimation first in cases without interference and then in ones affected by it. This thesis contains three main components: (1) a doubly robust method to estimate the optimal DTRs for individuals where the treatments of their connected neighbours in the same social network are taken into account in the decision rules; (2) a doubly robust method to estimate the optimal DTRs for binary outcomes using sequential weighted generalized linear models; (3) a doubly robust method to estimate the optimal DTRs for ordinal outcomes in the presence of household interference. In (1), we study the DTR estimation method of dynamic weighted ordinary least squares (dWOLS), which boasts easy implementation and double robustness, but relies on the no interference assumption. We define a network propensity function and build on it to establish an implementation of dWOLS that remains doubly robust under interference associated with network links. The method's properties are shown via simulation and applied to household pairs data from the Population Assessment of Tobacco and Health (PATH) Study. On the basis of the theories of dWOLS and using our interference-aware version, we focus on developing innovative DTR estimation methods for both binary and ordinal outcomes, in particular, the methods in the presence of interference. In (2), considering binary outcomes, we propose a new method for DTR estimation without interference, the dynamic weighted generalized linear model (dWGLM), which accommodates binary outcomes while offering relatively straightforward implementation and robustness to model misspecification. We introduce the method and its underlying theory, and illustrate both in an analysis of e-cigarette usage and smoking cessation, using the observational data from the PATH study. Finally, in (3), we further extend these regression-based DTR methods to the ordinal outcome case, and also propose a robust method — the dynamic weighted proportional odds model (dWPOM). Moreover, in the presence of household interference, exploring the possible correlation between treatments in the same household, we investigate the covariate balancing weights, which rely on the joint propensity score, and methods for estimating the joint propensity score. Examining different types of balancing weights, we verify the double robustness of dWPOM with our adjusted weights via simulation studies. Lastly, we also illustrate dWPOM in the analysis of data from PATH. For each participant's household, we derive the household treatment configuration recommendations for achieving the best outcome of the pair: both individuals quit or attempt to quit smoking

Automatic lexicon acquisition from encyclopedia.

Lo, Ka Kan.Thesis (M.Phil.)--Chinese University of Hong Kong, 2007.Includes bibliographical references (leaves 97-104).Abstracts in English and Chinese.Chapter 1 --- Introduction --- p.1Chapter 1.1 --- Motivation --- p.3Chapter 1.2 --- New paradigm in language learning --- p.5Chapter 1.3 --- Semantic Relations --- p.7Chapter 1.4 --- Contribution of this thesis --- p.9Chapter 2 --- Related Work --- p.13Chapter 2.1 --- Theoretical Linguistics --- p.13Chapter 2.1.1 --- Overview --- p.13Chapter 2.1.2 --- Analysis --- p.15Chapter 2.2 --- Computational Linguistics - General Learning --- p.17Chapter 2.3 --- Computational Linguistics - HPSG Lexical Acquisition --- p.20Chapter 2.4 --- Learning approach --- p.22Chapter 3 --- Background --- p.25Chapter 3.1 --- Modeling primitives --- p.26Chapter 3.1.1 --- Feature Structure --- p.26Chapter 3.1.2 --- Word --- p.28Chapter 3.1.3 --- Phrase --- p.35Chapter 3.1.4 --- Clause --- p.36Chapter 3.2 --- Wikipedia Resource --- p.38Chapter 3.2.1 --- Encyclopedia Text --- p.40Chapter 3.3 --- Semantic Relations --- p.40Chapter 4 --- Learning Framework - Syntactic and Semantic --- p.46Chapter 4.1 --- Type feature scoring function --- p.48Chapter 4.2 --- Confidence score of lexical entry --- p.50Chapter 4.3 --- Specialization and Generalization --- p.52Chapter 4.3.1 --- Further Processing --- p.54Chapter 4.3.2 --- Algorithm Outline --- p.54Chapter 4.3.3 --- Algorithm Analysis --- p.55Chapter 4.4 --- Semantic Information --- p.57Chapter 4.4.1 --- Extraction --- p.58Chapter 4.4.2 --- Induction --- p.60Chapter 4.4.3 --- Generalization --- p.63Chapter 4.5 --- Extension with new text documents --- p.65Chapter 4.6 --- Integrating the syntactic and semantic acquisition framework --- p.65Chapter 5 --- Evaluation --- p.68Chapter 5.1 --- Evaluation Metric - English Resource Grammar --- p.68Chapter 5.1.1 --- English Resource Grammar --- p.69Chapter 5.2 --- Experiments --- p.71Chapter 5.2.1 --- Tasks --- p.71Chapter 5.2.2 --- Evaluation Measures --- p.77Chapter 5.2.3 --- Methodologies --- p.78Chapter 5.2.4 --- Corpus Preparation --- p.79Chapter 5.2.5 --- Results --- p.81Chapter 5.3 --- Result Analysis --- p.85Chapter 6 --- Conclusions --- p.95Bibliography --- p.9

EG-ICE 2021 Workshop on Intelligent Computing in Engineering

The 28th EG-ICE International Workshop 2021 brings together international experts working at the interface between advanced computing and modern engineering challenges. Many engineering tasks require open-world resolutions to support multi-actor collaboration, coping with approximate models, providing effective engineer-computer interaction, search in multi-dimensional solution spaces, accommodating uncertainty, including specialist domain knowledge, performing sensor-data interpretation and dealing with incomplete knowledge. While results from computer science provide much initial support for resolution, adaptation is unavoidable and most importantly, feedback from addressing engineering challenges drives fundamental computer-science research. Competence and knowledge transfer goes both ways

EG-ICE 2021 Workshop on Intelligent Computing in Engineering

The 28th EG-ICE International Workshop 2021 brings together international experts working at the interface between advanced computing and modern engineering challenges. Many engineering tasks require open-world resolutions to support multi-actor collaboration, coping with approximate models, providing effective engineer-computer interaction, search in multi-dimensional solution spaces, accommodating uncertainty, including specialist domain knowledge, performing sensor-data interpretation and dealing with incomplete knowledge. While results from computer science provide much initial support for resolution, adaptation is unavoidable and most importantly, feedback from addressing engineering challenges drives fundamental computer-science research. Competence and knowledge transfer goes both ways

DEVELOPING MACHINE LEARNING METHODOLOGY FOR PRECISION HEALTH

Precision health has been an increasingly popular solution to improve health care quality and guide the decision making process. This includes precision medicine (at the individual level) and precision public health (at the population level such as communities and institutions). By learning from the available medical data with advanced analytical tools, precision health recommends the treatments that are individualized to each patient or entity to maximize clinical outcomes for each individual. We extend and develop three machine learning methods to improve the estimation of optimal individualized treatment regimes in precision health: the jackknife estimator of value function of precision medicine models compared with zero-order models, doubly robust outcome-weighted estimators with deep neural network structures for complex and large data, and risk-adjusted adverse event monitoring for survival data. First, motivated by a knee osteoarthristis trial, we estimate value functions and select the optimal treatment with the jackknife method whose consistency is established under weak assumptions. Next, we implement deep learning architecture in augmented outcome-weighted learning to increase model flexibility and computation efficiency, especially for high-dimensional data such as medical imaging. Lastly, we develop a risk-adjusted survival model to monitor adverse events and estimate its variance for hierarchical, right-censored data with recurrent events. All three methodologies aim to solve practical, health-related challenges and provide data-driven decision support and operations.Doctor of Philosoph

Novel Applications and Extensions for Bayesian Additive Regression Trees (BART) in Prediction, Imputation, and Causal Inference

The Bayesian additive regression trees (BART) is a method proposed by Chipman et al. (2010) that can handle non-linear main and multiple-way interaction effects for independent continuous or binary outcomes. It has enjoyed much success in areas like causal inference, economics, environmental sciences, and genomics. However, extensions of BART and application of these extensions are limited. This thesis discusses three novel applications and extensions for BART. We first discuss how BART can be extended to clustered outcomes by adding a random intercept. This work was motivated by the need to accurately predict driver behavior using observable speed and location information with application to communication of key human-driver intention to nearby vehicles in traffic. Although our extension can be considered a special case of the spatial BART (Zhang et al., 2007), our approach differs by providing a relatively simple algorithm that allows application to clustered binary outcomes. We next focus on the use of BART in missing data settings. Doubly robust (DR) methods allow consistent estimation of population means when either non-response propensity or modeling of the mean of the outcome is correctly specified. Kang and Schafer (2007) showed that DR methods produce biased and inefficient estimates when both propensity and mean models are misspecified. We consider the use of BART for modeling means and/or propensities to provide a ``robust-squared'' estimator that reduces bias and improves efficiency. We demonstrate this result, using simulations, for the two commonly used DR methods: Augmented Inverse Probability Weighting (AIPWT, Robbins et al., 1994) and penalized splines of propensity prediction (PSPP, Zhang and Little, 2009). We successfully applied our proposed model to two national crash datasets to impute missing change in deceleration values (delta-v) and missing Blood Alcohol Concentration (BAC) levels respectively. Our final effort considers how a negative wealth shock (sudden large decline in wealth) affects the cognitive outcome of late middle aged US adults using the Health Retirement Study, a longitudinal study of US adults, enrolled at age 50 and older and surveyed biennially since 1992. Our analysis faced three issues: lack of randomization, confounding by indication, and censoring of the cognitive outcome by a substantial number of deaths in our subjects. Marginal structural models (MSM), a commonly used method to deal with censoring by death, is arguably inappropriate because it upweights subjects who are more likely to die, creating a pseudo-population which resembles one where death is absent. We propose to compare the negative wealth shock effect only among subjects who survived under both sets of treatment regimens - a special case of principal stratification (Frangakis and Rubin, 2002). Because the counterfactual survival status would be unobserved, we imputed their survival status and restrict analysis to subjects who were observed and predicted to survive under both treatment regimes. We used a modified version of penalized spline of propensity methods in treatment comparisons (PENCOMP, Zhou et. al, 2018) to obtain a robust imputation of the counterfactual cognitive outcomes. Finally, we consider several possible extensions of these efforts for future work.PHDBiostatisticsUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/147594/1/vincetan_1.pd

Aeronautical engineering: A continuing bibliography with indexes (supplement 267)

This bibliography lists 661 reports, articles, and other documents introduced into the NASA scientific and technical information system in June, 1991. Subject coverage includes design, construction and testing of aircraft and aircraft engines; aircraft components, equipment and systems; ground support systems; theoretical and applied aspects of aerodynamics and general fluid dynamics; electrical engineering; aircraft control; remote sensing; computer sciences; nuclear physics; and social sciences