Meta-simulation for the Automated Design of Synthetic Overhead Imagery

The use of synthetic (or simulated) data for training machine learning models has grown rapidly in recent years. Synthetic data can often be generated much faster and more cheaply than its real-world counterpart. One challenge of using synthetic imagery however is scene design: e.g., the choice of content and its features and spatial arrangement. To be effective, this design must not only be realistic, but appropriate for the target domain, which (by assumption) is unlabeled. In this work, we propose an approach to automatically choose the design of synthetic imagery based upon unlabeled real-world imagery. Our approach, termed Neural-Adjoint Meta-Simulation (NAMS), builds upon the seminal recent meta-simulation approaches. In contrast to the current state-of-the-art methods, our approach can be pre-trained once offline, and then provides fast design inference for new target imagery. Using both synthetic and real-world problems, we show that NAMS infers synthetic designs that match both the in-domain and out-of-domain target imagery, and that training segmentation models with NAMS-designed imagery yields superior results compared to na\"ive randomized designs and state-of-the-art meta-simulation methods

A multifactorial analysis of FAP to regulate gastrointestinal cancers progression

BackgroundFibroblast activation protein (FAP) is a cell-surface serine protease that has both dipeptidyl peptidase as well as endopeptidase activities and could cleave substrates at post-proline bond. Previous findings showed that FAP was hard to be detected in normal tissues but significantly up-regulated in remodeling sites like fibrosis, atherosclerosis, arthritis and embryonic tissues. Though increasing evidence has demonstrated the importance of FAP in cancer progression, no multifactorial analysis has been developed to investigate its function in gastrointestinal cancers until now.MethodsBy comprehensive use of datasets from The Cancer Genome Atlas (TCGA), Clinical Proteomic Tumor Analysis Consortium (CPTAC), scTIME Portal and Human Protein Atlas (HPA), we evaluated the carcinogenesis potential of FAP in gastrointestinal cancers, analyzing the correlation between FAP and poor outcomes, immunology in liver, colon, pancreas as well as stomach cancers. Then liver cancer was selected as example to experimentally validate the pro-tumor and immune regulative role of FAP in gastrointestinal cancers.ResultsFAP was abundantly expressed in gastrointestinal cancers, such as LIHC, COAD, PAAD and STAD. Functional analysis indicated that the highly-expressed FAP in these cancers could affect extracellular matrix organization process and interacted with genes like COL1A1, COL1A2, COL3A1 and POSTN. In addition, it was also observed that FAP was positively correlated to M2 macrophages infiltration across these cancers. To verify these findings in vitro, we used LIHC as example and over-expressed FAP in human hepatic stellate LX2 cells, a main cell type that produce FAP in tumor tissues, and then investigate its role on LIHC cells as well as macrophages. Results showed that the medium from FAP-over-expressed LX2 cells could significantly promote the motility of MHCC97H and SK-Hep1 LIHC cells, increase the invasion of THP-1 macrophages and induce them into pro-tumor M2 phenotype.ConclusionIn summary, we employed bioinformatic tools and experiments to perform a comprehensive analysis about FAP. Up-regulation of FAP in gastrointestinal cancers was primarily expressed in fibroblasts and contributes to tumor cells motility, macrophages infiltration and M2 polarization, revealing the multifactorial role of FAP in gastrointestinal cancers progression

Synthetic Imagery Data Generation for Training and Testing Deep Learning Models of Object Recognition

A well-known challenge of modern high-capacity recognition models, such as deep neural networks, is their need for large quantities of real-world data collected for training and testing purposes. One potential solution to this problem is the use of synthetic data, either collected from a virtual world or generated from neural network models. The use of synthetic data to train or test recognition models has grown rapidly in recent years, and it has been found to be effective for a variety of tasks. My research investigates the utility of synthetic data for training and testing deep learning models using three classes of design strategies: i) image stylization, ii) domain randomization and iii) meta learning.My first set of work focuses on utilization of synthetic data generated from image stylization methods for testing of autonomous driving perception systems. Autonomous driving perception systems require robust testing that includes many abnormal conditions (e.g., severe weather conditions) to ensure safety. Such necessary validation requires large amounts of testing data collected under abnormal conditions, i.e., those that rarely occur, which is expensive to obtain in practice. Industry guiding the development of autonomous driving systems has collected large quantities of images obtained under normal driving conditions after years of on-road testing, however, the datasets of images that have been obtained under abnormal conditions are still insufficient for accurate validation of the driving algorithms. In my work, generative adversarial networks are trained to stylize the existing real-world images to reflect abnormal conditions. Novel adversarial training techniques are proposed to reduce the sim-to-real gap and ensure the imagery fidelity. The stylized synthetic images are applied to the testing of object detection models for autonomous vehicles under abnormal conditions, e.g., heavy fog density and low light intensity. The experimental results reveal the capability of generative adversarial networks to generate high-fidelity synthetic images with important characteristics of images collected in the abnormal conditions, similar to real images. The proposed data synthesis method can benefit car companies by reducing costs and time associated with the necessary testing under abnormal conditions. A domain randomization method is proposed in my second set of work for building detection in overhead imagery applications, where the available real-world data is extremely insufficient. The visual characteristics of overhead imagery vary tremendously due to numerous factors: imaging conditions, environmental conditions, and geographic location. To train a robust recognition model, a large set of training data in the target domain is required, however, it is extremely expensive to obtain such data in real applications, due to the cost of imagery, and the required manual pixel-wise labeling of the imagery. In my work, an efficient framework is proposed for automatic generation of large quantities of synthetic overhead imagery from virtual world simulators with randomized content and design parameters. Then, it is assumed that the simulation design space is large enough that the target domain is likely to be a subset of the simulation design space, borrowing the key hypothesis of domain randomization. The synthetic images are used for augmenting the training of building detection models in overhead imagery. The experimental results show that virtual world simulators have the capability to generate synthetic images with important characteristics of real images and benefit the training of building detection models, which confirms the hypothesis of domain randomization. To continue this path of research, I next investigated meta learning methods in my third effort for a more efficient data synthesis approach. Since domain randomization methods typically incur a long training period and lead to models that are too conservative, I propose a meta learning-based method, called Neural-Adjoint Meta-Simulation (NAMS), to learn how to accurately generate synthetic data in the target domain from virtual world simulators. Fast design parameter inferences of target data are learned based on a novel Neural-Adjoint technique which makes the optimization differentiable. Synthetic data from the target domain can then be efficiently generated based on the fast inferred design parameters. I show in several synthetic and real-world experiments that, by using NAMS, synthetic images with the target design can be rapidly and accurately generated for the training of building detection algorithms. The proposed method is amortized: after an upfront computational cost at the learning stage, parameters of new target data can be inferred efficiently. To my knowledge, this is the first work in the data synthesis area that emphasizes the amortization property, for an important fact in real applications, that is, the number of unique target domains are typically large.</p

Spatial Variation in Trophic Structure of Dominant Fish Species in Lake Dongting, China during Dry Season

Understanding trophic interactions in food webs is crucial to revealing the transfer of substances and energy from primary food sources to consumers in aquatic ecosystems. We hypothesize that the trophic structure of consumers can be significantly affected by primary food sources (pelagic, benthic, and littoral sources) through complex trophic interactions. This study used stable isotope analysis and Bayesian mixing models to estimate the trophic levels of fish consumers and the contributions of primary food sources in the three sub-lakes (Eastern, Southern, and Western Dongting) of Lake Dongting, which have different physical and chemical parameters of water, fish species diversity, and plankton (phytoplankton and zooplankton) density. Results showed the differences in community structures of fish among sub-lakes. Fish trophic levels were significantly higher in Eastern Dongting than those in the two other areas. The contributions of primary food sources to fishes were as follows: the pelagic source was the main basal food source in Eastern Dongting, and littoral and pelagic sources played equally essential roles in Southern Dongting; fishes in Western Dongting relied on more benthic source to growth than those in the two other regions. This study can fill gaps in our knowledge of the influence of the underlying food available on trophic structure of consumers by exploring the role of primary food sources and making the trophic structure of consumers in the aquatic food web highly complicated and diverse through control of the distribution of primary food sources

Photoacoustic Tomography with Temporal Encoding Reconstruction (PATTERN) for cross-modal individual analysis of the whole brain

Abstract Cross-modal analysis of the same whole brain is an ideal strategy to uncover brain function and dysfunction. However, it remains challenging due to the slow speed and destructiveness of traditional whole-brain optical imaging techniques. Here we develop a new platform, termed Photoacoustic Tomography with Temporal Encoding Reconstruction (PATTERN), for non-destructive, high-speed, 3D imaging of ex vivo rodent, ferret, and non-human primate brains. Using an optimally designed image acquisition scheme and an accompanying machine-learning algorithm, PATTERN extracts signals of genetically-encoded probes from photobleaching-based temporal modulation and enables reliable visualization of neural projection in the whole central nervous system with 3D isotropic resolution. Without structural and biological perturbation to the sample, PATTERN can be combined with other whole-brain imaging modalities to acquire the whole-brain image with both high resolution and morphological fidelity. Furthermore, cross-modal transcriptome analysis of an individual brain is achieved by PATTERN imaging. Together, PATTERN provides a compatible and versatile strategy for brain-wide cross-modal analysis at the individual level