Automating the conversion of natural language fiction to multi-modal 3D animated virtual environments

Popular fiction books describe rich visual environments that contain characters, objects, and behaviour. This research develops automated processes for converting text sourced from fiction books into animated virtual environments and multi-modal films. This involves the analysis of unrestricted natural language fiction to identify appropriate visual descriptions, and the interpretation of the identified descriptions for constructing animated 3D virtual environments. The goal of the text analysis stage is the creation of annotated fiction text, which identifies visual descriptions in a structured manner. A hierarchical rule-based learning system is created that induces patterns from example annotations provided by a human, and uses these for the creation of additional annotations. Patterns are expressed as tree structures that abstract the input text on different levels according to structural (token, sentence) and syntactic (parts-of-speech, syntactic function) categories. Patterns are generalized using pair-wise merging, where dissimilar sub-trees are replaced with wild-cards. The result is a small set of generalized patterns that are able to create correct annotations. A set of generalized patterns represents a model of an annotator's mental process regarding a particular annotation category. Annotated text is interpreted automatically for constructing detailed scene descriptions. This includes identifying which scenes to visualize, and identifying the contents and behaviour in each scene. Entity behaviour in a 3D virtual environment is formulated using time-based constraints that are automatically derived from annotations. Constraints are expressed as non-linear symbolic functions that restrict the trajectories of a pair of entities over a continuous interval of time. Solutions to these constraints specify precise behaviour. We create an innovative quantified constraint optimizer for locating sound solutions, which uses interval arithmetic for treating time and space as contiguous quantities. This optimization method uses a technique of constraint relaxation and tightening that allows solution approximations to be located where constraint systems are inconsistent (an ability not previously explored in interval-based quantified constraint solving). 3D virtual environments are populated by automatically selecting geometric models or procedural geometry-creation methods from a library. 3D models are animated according to trajectories derived from constraint solutions. The final animated film is sequenced using a range of modalities including animated 3D graphics, textual subtitles, audio narrations, and foleys. Hierarchical rule-based learning is evaluated over a range of annotation categories. Models are induced for different categories of annotation without modifying the core learning algorithms, and these models are shown to be applicable to different types of books. Models are induced automatically with accuracies ranging between 51.4% and 90.4%, depending on the category. We show that models are refined if further examples are provided, and this supports a boot-strapping process for training the learning mechanism. The task of interpreting annotated fiction text and populating 3D virtual environments is successfully automated using our described techniques. Detailed scene descriptions are created accurately, where between 83% and 96% of the automatically generated descriptions require no manual modification (depending on the type of description). The interval-based quantified constraint optimizer fully automates the behaviour specification process. Sample animated multi-modal 3D films are created using extracts from fiction books that are unrestricted in terms of complexity or subject matter (unlike existing text-to-graphics systems). These examples demonstrate that: behaviour is visualized that corresponds to the descriptions in the original text; appropriate geometry is selected (or created) for visualizing entities in each scene; sequences of scenes are created for a film-like presentation of the story; and that multiple modalities are combined to create a coherent multi-modal representation of the fiction text. This research demonstrates that visual descriptions in fiction text can be automatically identified, and that these descriptions can be converted into corresponding animated virtual environments. Unlike existing text-to-graphics systems, we describe techniques that function over unrestricted natural language text and perform the conversion process without the need for manually constructed repositories of world knowledge. This enables the rapid production of animated 3D virtual environments, allowing the human designer to focus on creative aspects

GENERIC FRAMEWORKS FOR INTERACTIVE PERSONALIZED INTERESTING PATTERN DISCOVERY

The traditional frequent pattern mining algorithms generate an exponentially large number of patterns of which a substantial portion are not much significant for many data analysis endeavours. Due to this, the discovery of a small number of interesting patterns from the exponentially large number of frequent patterns according to a particular user\u27s interest is an important task. Existing works on patter

A Survey on Event-based News Narrative Extraction

Narratives are fundamental to our understanding of the world, providing us with a natural structure for knowledge representation over time. Computational narrative extraction is a subfield of artificial intelligence that makes heavy use of information retrieval and natural language processing techniques. Despite the importance of computational narrative extraction, relatively little scholarly work exists on synthesizing previous research and strategizing future research in the area. In particular, this article focuses on extracting news narratives from an event-centric perspective. Extracting narratives from news data has multiple applications in understanding the evolving information landscape. This survey presents an extensive study of research in the area of event-based news narrative extraction. In particular, we screened over 900 articles that yielded 54 relevant articles. These articles are synthesized and organized by representation model, extraction criteria, and evaluation approaches. Based on the reviewed studies, we identify recent trends, open challenges, and potential research lines.Comment: 37 pages, 3 figures, to be published in the journal ACM CSU

Development of a diffusion kernel density estimator and application on marine carbon-13 isotope data

My work developed a kernel density estimator that well resolves typical structures of probability densities, which was demonstrated on a newly compiled marine data set of organic carbon-13 isotope ratios (δ13CPOC). All work was conducted within the emerging field of marine data science. I identified classical data science, a general understanding of ocean science, communication skills, and confidence as requirements for marine data scientists. In the beginning of my work, the existing δ13CPOC data consisted of about 500 data points in the global ocean. I expanded the existing data set to 4732 data points in a first version, and additionally to 6952 in a second. Both are published at PANGAEA along with meta information such as measurement location, time, and method, and interpolations. I have published a description of the temporal and geographic distribution of the first version at Earth System Science Data. I designed the development of the kernel density estimator algorithm on the existing concept of computing it as a solution of the diffusion equation. My algorithm uses finite differences in space and equidistant time steps with an implicit Euler method, and approximates the optimal smoothing parameter by two pilot steps. Compared to other well-known kernel density estimators, my algorithm produces reliable approximations of multimodal and boundary-close distributions on artificial and real marine data and is robust to noise. My implementation is published as a Python package on Zenodo, its description is submitted to Geoscientific Model Development. I was able to show that my kernel density estimator reliably evalu- ates ocean data and thus lays a foundation for calibrating Earth system models. At the same time, I was able to contribute to the definition and establishment of the field of Marine Data Science

An approach to the visualization of adaptive hypermedia structures and other small-world networks based on hierarchically Clustered Graphs

Tesis doctoral inédita. Universidad Autónoma de Madrid, Escuela Politécnica Superior, julio de 200

Tune your brown clustering, please

Brown clustering, an unsupervised hierarchical clustering technique based on ngram mutual information, has proven useful in many NLP applications. However, most uses of Brown clustering employ the same default configuration; the appropriateness of this configuration has gone predominantly unexplored. Accordingly, we present information for practitioners on the behaviour of Brown clustering in order to assist hyper-parametre tuning, in the form of a theoretical model of Brown clustering utility. This model is then evaluated empirically in two sequence labelling tasks over two text types. We explore the dynamic between the input corpus size, chosen number of classes, and quality of the resulting clusters, which has an impact for any approach using Brown clustering. In every scenario that we examine, our results reveal that the values most commonly used for the clustering are sub-optimal

Anomaly Detection With Machine Learning In Astronomical Images

Masters of ScienceObservations that push the boundaries have historically fuelled scientific breakthroughs, and these observations frequently involve phenomena that were previously unseen and unidentified. Data sets have increased in size and quality as modern technology advances at a record pace. Finding these elusive phenomena within these large data sets becomes a tougher challenge with each advancement made. Fortunately, machine learning techniques have proven to be extremely valuable in detecting outliers within data sets. Astronomaly is a framework that utilises machine learning techniques for anomaly detection in astronomy and incorporates active learning to provide target specific results. It is used here to evaluate whether machine learning techniques are suitable to detect anomalies within the optical astronomical data obtained from the Dark Energy Camera Legacy Survey. Using the machine learning algorithm isolation forest, Astronomaly is applied on subsets of the Dark Energy Camera Legacy Survey (DECaLS) data set. The pre-processing stage of Astronomaly had to be significantly extended to handle real survey data from DECaLS, with the changes made resulting in up to 10% more sources having their features extracted successfully. For the top 500 sources returned, 292 were ordinary sources, 86 artefacts and masked sources and 122 were interesting anomalous sources. A supplementary machine learning algorithm known as active learning enhances the identification probability of outliers in data sets by making it easier to identify target specific sources. The addition of active learning further increases the amount of interesting sources returned by almost 40%, with 273 ordinary sources, 56 artefacts and 171 interesting anomalous sources returned. Among the anomalies discovered are some merger events that have been successfully identified in known catalogues and several candidate merger events that have not yet been identified in the literature. The results indicate that machine learning, in combination with active learning, can be effective in detecting anomalies in actual data sets. The extensions integrated into Astronomaly pave the way for its application on future surveys like the Vera C. Rubin Observatory Legacy Survey of Space and Time

Knowledge Transfer in Object Recognition.

PhD Thesis.Abstract Object recognition is a fundamental and long-standing problem in computer vision. Since the latest resurgence of deep learning, thousands of techniques have been proposed and brought to commercial products to facilitate people’s daily life. Although remarkable achievements in object recognition have been witnessed, existing machine learning approaches remain far away from human vision system, especially in learning new concepts and Knowledge Transfer (KT) across scenarios. One main reason is that current learning approaches address isolated tasks by independently training predefined models, without considering any knowledge learned from previous tasks or models. In contrast, humans have an inherent ability to transfer the knowledge acquired from earlier tasks or people to new scenarios. Therefore, to scaling object recognition in realistic deployment, effective KT schemes are required. This thesis studies several aspects of KT for scaling object recognition systems. Specifically, to facilitate the KT process, several mechanisms on fine-grained and coarse-grained object recognition tasks are analyzed and studied, including 1) cross-class KT on person re-identification (reid); 2) cross-domain KT on person re-identification; 3) cross-model KT on image classification; 4) cross-task KT on image classification. In summary, four types of knowledge transfer schemes are discussed as follows: Chapter 3 Cross-class KT in person re-identification, one of representative fine-grained object recognition tasks, is firstly investigated. The nature of person identity classes for person re-id are totally disjoint between training and testing (a zero-shot learning problem), resulting in the highly demand of cross-class KT. To solve that, existing person re-id approaches aim to derive a feature representation for pairwise similarity based matching and ranking, which is able to generalise to test. However, current person re-id methods assume the provision of accurately cropped person bounding boxes and each of them is in the same resolution, ignoring the impact of the background noise and variant scale of images to cross-class KT. This is more severed in practice when person bounding boxes must be detected automatically given a very large number of images and/or videos (un-constrained scene images) processed. To address these challenges, this chapter provides two novel approaches, aiming to promote cross-class KT and boost re-id performance. 1) This chapter alleviates inaccurate person bounding box by developing a joint learning deep model that optimises person re-id attention selection within any auto-detected person bounding boxes by reinforcement learning of background clutter minimisation. Specifically, this chapter formulates a novel unified re-id architecture called Identity DiscriminativE Attention reinforcement Learning (IDEAL) to accurately select re-id attention in auto-detected bounding boxes for optimising re-id performance. 2) This chapter addresses multi-scale problem by proposing a Cross-Level Semantic Alignment (CLSA) deep learning approach capable of learning more discriminative identity feature representations in a unified end-to-end model. This 4 is realised by exploiting the in-network feature pyramid structure of a deep neural network enhanced by a novel cross pyramid-level semantic alignment loss function. Extensive experiments show the modelling advantages and performance superiority of both IDEAL and CLSA over the state-of-the-art re-id methods on widely used benchmarking datasets. Chapter 4 In this chapter, we address the problem of cross-domain KT in unsupervised domain adaptation for person re-id. Specifically, this chapter considers cross-domain KT as follows: 1) Unsupervised domain adaptation: “train once, run once” pattern, transferring knowledge from source domain to specific target domain and the model is restricted to be applied on target domain only; 2) Universal re-id: “train once, run everywhere” pattern, transferring knowledge from source domain to any target domains, and therefore is capable of deploying any domains of re-id task. This chapter firstly develops a novel Hierarchical Unsupervised Domain Adaptation (HUDA) method for unsupervised domain adaptation for re-id. It can automatically transfer labelled information of an existing dataset (a source domain) to an unlabelled target domain for unsupervised person re-id. Specifically, HUDA is designed to model jointly global distribution alignment and local instance alignment in a two-level hierarchy for discovering transferable source knowledge in unsupervised domain adaptation. Crucially, this approach aims to overcome the under-constrained learning problem of existing unsupervised domain adaptation methods, lacking of the local instance alignment constraint. The consequence is more effective and cross-domain KT from the labelled source domain to the unlabelled target domain. This chapter further addresses the limitation of “train once, run once ” for existing domain adaptation person re-id approaches by presenting a novel “train once, run everywhere” pattern. This conventional “train once, run once” pattern is unscalable to a large number of target domains typically encountered in real-world deployments, due to the requirement of training a separate model for each target domain as supervised learning methods. To mitigate this weakness, a novel “Universal Model Learning” (UML) approach is formulated to enable domain-generic person re-id using only limited training data of a “single” seed domain. Specifically, UML trains a universal re-id model to discriminate between a set of transformed person identity classes. Each of such classes is formed by applying a variety of random appearance transformations to the images of that class, where the transformations simulate camera viewing conditions of any domains for making the model domain generic. Chapter 5 The third problem considered in this thesis is cross-model KT in coarse-grained object recognition. This chapter discusses knowledge distillation in image classification. Knowledge distillation is an effective approach to transfer knowledge from a large teacher neural network to a small student (target) network for satisfying the low-memory and fast running requirements. Whilst being able to create stronger target networks compared to the vanilla non-teacher based learning strategy, this scheme needs to train additionally a large teacher model with expensive computational cost and requires complex multi-stages training. This chapter firstly presents a Self-Referenced Deep Learning (SRDL) strategy to accelerate the training process. Unlike both vanilla optimisation and knowledge distillation, SRDL distils the knowledge discovered by the in-training target model back to itself for regularising the subsequent learning procedure therefore eliminating the need for training a large teacher model. Secondly, an On-the-fly Native Ensemble (ONE) learning strategy for one-stage knowledge distillation is proposed to solve the weakness of complex multi-stages training. Specifically, ONE only trains a single multi-branch network while simultaneously establishing a strong teacher on-the-fly to enhance the learning of target network. Chapter 6 Forth, this thesis studies the cross-task KT in coarse-grained object recognition. This chapter focuses on the few-shot classification problem, which aims to train models capable of recognising new, previously unseen categories from the novel task by using only limited training samples. Existing metric learning approaches constitute a highly popular strategy, learning discriminative representations such that images, containing different classes, are well separated in an embedding space. The commonly held assumption that each class is summarised by a sin5 gle, global representation (referred to as a prototype) that is then used as a reference to infer class labels brings significant drawbacks. This formulation fails to capture the complex multi-modal latent distributions that often exist in real-world problems, and yields models that are highly sensitive to the prototype quality. To address these limitations, this chapter proposes a novel Mixture of Prototypes (MP) approach that learns multi-modal class representations, and can be integrated into existing metric based methods. MP models class prototypes as a group of feature representations carefully designed to be highly diverse and maximise ensembling performance. Furthermore, this thesis investigates the benefit of incorporating unlabelled data in cross-task KT, and focuses on the problem of Semi-Supervised Few-shot Learning (SS-FSL). Recent SSFSL work has relied on popular Semi-Supervised Learning (SSL) concepts, involving iterative pseudo-labelling, yet often yields models that are susceptible to error propagation and sensitive to initialisation. To address this limitation, this chapter introduces a novel prototype-based approach (Fewmatch) for SS-FSL that exploits model Consistency Regularization (CR) in a robust manner and promotes cross-task unlabelled data knowledge transfer. Fewmatch exploits unlabelled data via Dynamic Prototype Refinement (DPR) approach, where novel class prototypes are alternatively refined 1) explicitly, using unlabelled data with high confidence class predictions and 2) implicitly, by model fine-tuning using a data selective model CR loss. DPR affords CR convergence, with the explicit refinement providing an increasingly stronger initialisation and alleviates the issue of error propagation, due to the application of CR. Chapter 7 draws conclusions and suggests future works that extend the ideas and methods developed in this thesi

Phytosterols and blood lipid risk factors for cardiovascular disease

Cardiovascular disease (CVD) is the leading cause of morbidity and mortality worldwide. Lifestyle improvements including dietary changes are important for CVD prevention. This thesis aimed to advance insights in the role of phytosterols, lipid-like compounds present in foods or plant origin, in the management of blood lipid risk factors for CVD. Phytosterols include plant sterols and their saturated form, plant stanols. These compounds resemble cholesterol in both structure and function, but cannot be produced by the human body. The intake of phytosterols occurs through plant-based foods and/or enriched foods like margarine. Elevated blood low-density lipoprotein cholesterol (LDL-C) is a major risk factor for CVD, especially for coronary heart disease (CHD) resulting from atherosclerosis. We studied the dose-response relationship between dietary phytosterols and blood LDL-C in two meta-analyses (Chapters 2 and 3). A meta-analysis of 81 randomized controlled trials (Chapter 2) demonstrated a non-linear, continuous dose-response relationship for the LDL-C-lowering effect of phytosterols. Based on this dose-response curve, it may be predicted that phytosterols at a dose of 2 g/d lower LDL-C by 0.35 mmol/L or 9%. The dose-response curve reached a plateau at phytosterol doses of ~3 g/d, above which there is limited additional LDL-C-lowering effect. In another meta-analysis of 124 randomized controlled trials (Chapter 3), we showed that plant sterols and plant stanols up to ~3 g/d are equally effective in lowering LDL-C by a maximum of 12%. No conclusions could be drawn for phytosterol doses exceeding 4 g/d because of the limited number of studies. Elevated blood triglycerides (TGs) may also be involved in the onset of CVD, although its role is less established than for LDL-C. The effect of plant sterols on blood TG concentrations was assessed in a meta-analysis of individual subject data from 12 randomized controlled trials (Chapter 4). We showed that plant sterols, at a dose of ~2 g/d, modestly reduce TG concentrations by on average 0.12 mmol/L or 6%. The TG-lowering effect of plant sterols was larger in subjects with higher initial TG concentrations. Our double-blind, placebo-controlled, randomized trial with 332 subjects (Chapter 5) showed more pronounced TG-lowering effects of 9-16% when plant sterols (2.5 g/d) were combined with low doses of omega-3 fish fatty acids (0.9 to 1.8 g/d). Dietary phytosterols are, after initial absorption by intestinal cells, actively excreted back into the intestinal lumen. Nevertheless, small amounts reach the circulation. We assessed the effect of plant sterol intake on blood plant sterol concentrations in a meta-analysis of 41 randomized controlled trials (Chapter 6). The intake of plant sterols, at a dose of ~1.6 g/d, increased blood sitosterol concentrations by on average 2 μmol/L (31%) and campesterol concentrations by 5 μmol/L (37%). At the same time, total cholesterol and LDL-C concentrations were reduced by on average 0.36 mmol/L (6%) and 0.33 mmol/L (9%), respectively. After supplemental intake, plant sterol concentrations remained below 1% of total sterols circulating in the blood. Whether phytosterols, due to their LDL-C-lowering properties, affect the risk of CVD events is at present unknown. The relation between phytosterol intake from natural sources (e.g. vegetables, cereals, nuts) and CVD risk in the population was examined in a large prospective cohort of 35,597 Dutch men and women with 12 years of follow-up (Chapter 7). The intake of phytosterols from natural sources (~300 mg/d) was not related to risk of CVD (total of 3,047 events) with a relative risk ranging from 0.90 to 0.99 across quintiles of phytosterol intake. Also, no association with incident CHD and myocardial infarction were found. In a cross-sectional analysis using baseline data of this cohort, phytosterol intake was associated with lower blood LDL-C in men (-0.18 mmol/L per 50 mg/d; 95% CI: -0.29; -0.08) but not in women (-0.03 mmol/L; 95% CI: -0.08; 0.03). Most randomized trials with enriched foods have tested phytosterol doses between 1.5 and 2.4 g/d. In practice, however, users of such foods consume much lower amounts (~1 g/d), which is about 3 times higher than obtained from a regular Western diet. Individuals who consume diets with emphasis on plant-based foods (e.g. vegetarians) may reach phytosterol intakes between 0.5 and 1 g/d. Health authorities recommend various types of diets for CVD prevention, almost all rich in plant-based foods and, consequently, relatively rich in phytosterols. In conclusion, a high intake of phytosterols with enriched foods was shown to lower LDL-C in a dose-dependent manner. Furthermore, a high intake of plant sterols with enriched foods modestly lowered TG concentrations and increased plasma plant sterol concentrations. A low intake of naturally occurring phytosterols in the general population did not show a clear association with CVD risk. Based on these findings, the intake of phytosterols may be considered in the management of hypercholesterolemia. Whether a high intake of phytosterols can play a role in CVD prevention in the population at large remains to be established.</p