Exploiting Intrinsic Clustering Structure in Discrete-Valued Data Sets for Efficient Knowledge Discovery in the Presence of Missing Data

Scalable algorithm design has become central in the era of large-scale data analysis. The vast amounts of data pouring in from a diverse set of application domains, such as bioinformatics, recommender systems, sensor systems, and social networks, cannot be analyzed efficiently using many data mining and statistical tools that were designed for a small scale setting. It is an ongoing challenge to the data mining, machine learning, and statistics communities to design new methods for efficient data analysis. Confounding this challenge is the noisy and incomplete nature of real-world data sets. Research scientists as well as practitioners in industry need to find meaningful patterns in data with missing value rates often as high as 99%, in addition to errors in the data that can obstruct accurate analyses. My contribution to this line of research is the design of new algorithms for scalable clustering, data reduction, and similarity evaluation by exploiting inherent clustering structure in the input data to overcome the challenges of significant amounts of missing entries. I demonstrate that, by focusing on underlying clustering properties of the data, we can improve the efficiency of several data analysis methods on sparse, discrete-valued data sets. I highlight new methods that I have developed with my collaborators for three diverse knowledge discovery tasks: (1) clustering genetic markers into linkage groups, (2) reducing large-scale genetic data to a much smaller, more accurate representative data set, and (3) computing similarity between users in recommender systems. In each case, I point out how the underlying clustering structure can be used to design more efficient algorithms, even when high missing value rates are present

Feinkartierung von Quantitative Trait Loci für Somatische Zellzahl auf Bos taurus Autosomen 2, 18 und 27 in der Deutsche Holstein

The objectives of this thesis were two-fold. Firstly, software was developed to properly analyse data collected in the context of a national genome project (FUGATO M.A.S.-Net). Secondly, analyses were carried out for data collected from M.A.S.-Net partner laboratories in Giessen, Kiel and Dummerstorf. Phenotypic values for somatic cell score of German Holsteins were contributed by the animal computing center (VIT) in Verden.Ziel dieser Arbeit war es, eine Software zu entwickeln, die die Analyse von Markerinformationen anhand von Varianzkomponentenmethoden und Haplotyp-Effekten möglich macht. Ebenfalls sollten die Daten, die im Rahmen des FUGATO M.A.S.-Net Projektes gesammelt wurden und aus den Partnerlabors in Gießen, Kiel und Dummerstorf stammen, analysiert werden. Die Phänotypen wurden vom VIT (Vereinigte Informationssysteme Tierhaltung) in Verden zur Verfügung gestellt

Using mapped quantitative trait loci in improving genetic evaluation

The benefit of using QTL information in dairy cattle breeding schemes by means of computer simulation is investigated. In addition, algorithms to overcome computational problems arising when marker data are included in mixed linear models were proposed.;Computer simulation was conducted with parameters relative to the Holstein population of the United States. Superiority of QTL-assisted selection (QAS) over QTL-free selection was studied in four pathways of selection, namely active sires, young bulls, bull dams, and cows, for cumulative genetic response, accuracy of evaluation, and selection pressure on the QTL.;Further, breeding scheme as a factor was studied. The breeding scheme was the most effective factor in increasing the superiority of QAS. As it agreed with many previous studies, nucleus breeding schemes were found to be promising systems to implement QTL information. On the other hand, benefits of QAS in conventional two stage selection programs were limited.;The interaction between the type of QTL information available and the breeding system was found important. Using a highly polymorphic QTL in nucleus schemes was found very effective. Effects of different number of alleles per locus and different number of loci on the superiority of QAS were studied.;An algorithm to directly build the inverse of a conditional gametic relationship matrix, given marker data, was developed. The inverse algorithm is based on matrix decomposition instead of partitioned matrix theory. Numerical techniques that greatly improved computing performance were introduced.;Appropriate modifications to the conventional breeding schemes that are currently in use are highly recommended. Further, attention should be paid to the characteristics of the QTL and how they may interact with the breeding system, e.g., number of loci and alleles. Finally, the study found that the use of marked or known QTL information in genetic evaluation is computationally possible and generally useful

Neurospora Bibliography

This bibliography represents my attempt to collect all works dealing substantially with Neurospora

Biologically Interpretable, Integrative Deep Learning for Cancer Survival Analysis

Identifying complex biological processes associated to patients\u27 survival time at the cellular and molecular level is critical not only for developing new treatments for patients but also for accurate survival prediction. However, highly nonlinear and high-dimension, low-sample size (HDLSS) data cause computational challenges in survival analysis. We developed a novel family of pathway-based, sparse deep neural networks (PASNet) for cancer survival analysis. PASNet family is a biologically interpretable neural network model where nodes in the network correspond to specific genes and pathways, while capturing nonlinear and hierarchical effects of biological pathways associated with certain clinical outcomes. Furthermore, integration of heterogeneous types of biological data from biospecimen holds promise of improving survival prediction and personalized therapies in cancer. Specifically, the integration of genomic data and histopathological images enhances survival predictions and personalized treatments in cancer study, while providing an in-depth understanding of genetic mechanisms and phenotypic patterns of cancer. Two proposed models will be introduced for integrating multi-omics data and pathological images, respectively. Each model in PASNet family was evaluated by comparing the performance of current cutting-edge models with The Cancer Genome Atlas (TCGA) cancer data. In the extensive experiments, PASNet family outperformed the benchmarking methods, and the outstanding performance was statistically assessed. More importantly, PASNet family showed the capability to interpret a multi-layered biological system. A number of biological literature in GBM supported the biological interpretation of the proposed models. The open-source software of PASNet family in PyTorch is publicly available at https://github.com/DataX-JieHao

Interaction Testing, Fault Location, and Anonymous Attribute-Based Authorization

abstract: This dissertation studies three classes of combinatorial arrays with practical applications in testing, measurement, and security. Covering arrays are widely studied in software and hardware testing to indicate the presence of faulty interactions. Locating arrays extend covering arrays to achieve identification of the interactions causing a fault by requiring additional conditions on how interactions are covered in rows. This dissertation introduces a new class, the anonymizing arrays, to guarantee a degree of anonymity by bounding the probability a particular row is identified by the interaction presented. Similarities among these arrays lead to common algorithmic techniques for their construction which this dissertation explores. Differences arising from their application domains lead to the unique features of each class, requiring tailoring the techniques to the specifics of each problem. One contribution of this work is a conditional expectation algorithm to build covering arrays via an intermediate combinatorial object. Conditional expectation efficiently finds intermediate-sized arrays that are particularly useful as ingredients for additional recursive algorithms. A cut-and-paste method creates large arrays from small ingredients. Performing transformations on the copies makes further improvements by reducing redundancy in the composed arrays and leads to fewer rows. This work contains the first algorithm for constructing locating arrays for general values of $d$ and $t$. A randomized computational search algorithmic framework verifies if a candidate array is $(\bar{d},t)$-locating by partitioning the search space and performs random resampling if a candidate fails. Algorithmic parameters determine which columns to resample and when to add additional rows to the candidate array. Additionally, analysis is conducted on the performance of the algorithmic parameters to provide guidance on how to tune parameters to prioritize speed, accuracy, or a combination of both. This work proposes anonymizing arrays as a class related to covering arrays with a higher coverage requirement and constraints. The algorithms for covering and locating arrays are tailored to anonymizing array construction. An additional property, homogeneity, is introduced to meet the needs of attribute-based authorization. Two metrics, local and global homogeneity, are designed to compare anonymizing arrays with the same parameters. Finally, a post-optimization approach reduces the homogeneity of an anonymizing array.Dissertation/ThesisDoctoral Dissertation Computer Science 201

Uncertainty Minimization in Robotic 3D Mapping Systems Operating in Dynamic Large-Scale Environments

This dissertation research is motivated by the potential and promise of 3D sensing technologies in safety and security applications. With specific focus on unmanned robotic mapping to aid clean-up of hazardous environments, under-vehicle inspection, automatic runway/pavement inspection and modeling of urban environments, we develop modular, multi-sensor, multi-modality robotic 3D imaging prototypes using localization/navigation hardware, laser range scanners and video cameras. While deploying our multi-modality complementary approach to pose and structure recovery in dynamic real-world operating conditions, we observe several data fusion issues that state-of-the-art methodologies are not able to handle. Different bounds on the noise model of heterogeneous sensors, the dynamism of the operating conditions and the interaction of the sensing mechanisms with the environment introduce situations where sensors can intermittently degenerate to accuracy levels lower than their design specification. This observation necessitates the derivation of methods to integrate multi-sensor data considering sensor conflict, performance degradation and potential failure during operation. Our work in this dissertation contributes the derivation of a fault-diagnosis framework inspired by information complexity theory to the data fusion literature. We implement the framework as opportunistic sensing intelligence that is able to evolve a belief policy on the sensors within the multi-agent 3D mapping systems to survive and counter concerns of failure in challenging operating conditions. The implementation of the information-theoretic framework, in addition to eliminating failed/non-functional sensors and avoiding catastrophic fusion, is able to minimize uncertainty during autonomous operation by adaptively deciding to fuse or choose believable sensors. We demonstrate our framework through experiments in multi-sensor robot state localization in large scale dynamic environments and vision-based 3D inference. Our modular hardware and software design of robotic imaging prototypes along with the opportunistic sensing intelligence provides significant improvements towards autonomous accurate photo-realistic 3D mapping and remote visualization of scenes for the motivating applications

Movement across scales: red fox spatial ecology

Rødrevens forflytningsmønster på forskjellige romlige skalaer Menneskelig påvirkning på naturlige habitater kan medføre reduserte bevegelsesmuligheter for noen dyrearter, eller fasilitere typiske generalister som rødrev Vulpes vulpes. Vår forståelse av rødrev i landskaper dominert av menneskelige aktiviteter er begrenset av manglende kunnskap om hvordan revene beveger seg i disse områdene. I denne avhandlingen undersøkte jeg temporale og romlige mønstre i forflytningene til rødrever langs en gradient av menneskelig påvirkning ved bruk av GPS-telemetridata innhentet i Norge og Sverige fra 2011 til 2019. Rødrevenes leveområder var langt større enn observert i tidligere studier, og med en markant individuell variasjon som delvis kunne relateres til miljøfaktorer langs en landskapsgradient. I lavereliggende sørlige områder med relativt høy primærproduksjon og mye landbruk var revenes leveområder fire ganger mindre enn i høyereliggende nordlige barskogområder. I gjennomsnitt var 43% av revenes posisjoner innenfor klustere som dekket bare 1% av leveområdene. Dette indikerte at rødrevenes kognitive kartlegging innebar repetitive bevegelser som bidro til avgrensing av leveområdene. Noen rever viste også en betydelig evne til å forflytte seg lange avstander under spredning, mellom populasjoner og landskaper, og potensielt over nasjonale grenser. Allikevel viste revene en genetisk struktur på liten romlig skala som var knyttet til sosiale mekanismer heller enn forflytningsevne og spredningskapasitet. Parvise distanser mellom nært beslektede hunndyr (gjennomsnitt = 6.3 km) var signifikant kortere enn avstandene mellom beslektede hanner (37.8 km). Dette understreker at sosiale forhold (som slektskap) spiller en viktig rolle i rødrevenes romlige organisasjon. Mine resultater viser at forståelsen av rødrevens forflytninger forutsetter god innsikt i hvilken temporal og romlig skala bevegelsene foregår. Å klassifisere unike og variable atferdstrekk hos en svært fleksibel art som rødrev er veldig vanskelig, og dette understreker artens økologiske plastisitet. Alt i alt har denne avhandlingen gitt ny innsikt i hvordan rødrevens forflytningsmønster er påvirket av sosial struktur og miljøfaktorer på forskjellige romlige skalaer. Dette har betydning for framtidig forskning og forvaltning, samt for modellering av revenes demografi og sykdomsspredning. Informasjonen gir økt innsikt i rødrevens områdebruk og spredning i rurale områder, og åpner for nye muligheter for forskning på denne artens påvirkning på økosystemer.Movement across scales: red fox spatial ecology The impact of human activities is altering natural habitats, reducing the ability of some animals to move, while facilitating other, generalist species, such as the red fox Vulpes vulpes. Our understanding of red foxes in rural and human modified landscapes is constrained by a lack of knowledge about how foxes use these landscapes. In this thesis I investigated the spatiotemporal movement patterns of red foxes along a landscape gradient of human influence using individual based GPS telemetry data from red foxes collected in Norway and Sweden between 2011- 2019. Herein, I identified much larger home ranges than previously recorded for red foxes and a high degree of individual variation, partially explained through environmental factors along a landscape gradient. At lower elevations, where productivity and the amount of available agricultural land increased, red foxes had home ranges approximately four times smaller than the home ranges of foxes in the northern boreal vegetation areas. I also identified cognitive mapping as a feature of red fox space use, linked to recursive movements within home ranges and contributing to bounded space use. On average, 43% of a red fox’s positions were found in defined clusters that covered a proportional area of only 1% of their home range. I highlighted the red fox’s ability to traverse between populations, across landscapes, and potentially across international boundaries, by identifying six long-distance dispersal events, representing some of the longest dispersal distances recorded for red foxes. However, I also showed that fine scale familial structuring in red foxes occurred by social mechanisms not linked to their movement ability or dispersal capacity. I found significant differences in pairwise geographic distances between highly related same sex pairs with the average distance between related males, 37.8 km, being six times farther than that of related females, averaging 6.3 km. This highlights how social dynamics (e.g. kin clustering and female philopatry) play a role in the spatial organization of red foxes. Finally, I showed that recognizing red fox behaviors is dependent on not only identifying their associated movement patterns, but also understanding the temporal and spatial scales at which their movements occur. Identifying the unique and variable behaviors of a highly flexible species such as the red fox is difficult and shows the ecological plasticity of the species. Together, this information represents new observations that greatly expand our knowledge of red fox space use and dispersal in rural landscapes and opens the door for future research into the broader ecosystem consequences of such movements. Overall, this thesis increases our understanding of red fox movement behaviors and their interactions with social and environmental factors at multiple spatial scales, with implications for future research, management and demographic and disease modeling.publishedVersio

Bayesian Model-based Methods for the Analysis of DNA Microarrays with Survival, Genetic, and Sequence Data

DNA microarrays measure the expression of thousands of genes or DNA fragments simultaneously in which probes have specific complementary hybridization. Gene expression or microarray data analysis problems have a prominent role in the biostatistics, biological sciences, and clinical medicine. The first paper proposes a method for finding associations between the survival time of the subjects and the gene expression of tumor microarrays. Measurement error is known to bias the estimates for survival regression coefficients, and this method minimizes bias. The latent variable model is shown to detect associations between potentially important genes and survival in a breast cancer dataset that conventional models did not detect, and the method is demonstrated to have robustness to misspecification with simulated data. The second paper considers the Expression Quantitative Trait Loci (eQTL) detection problem. An eQTL is a genetic locus that influences gene expression, and the major challenges with this type of data are multiple testing and computational issues. The proposed method extends the Mixture Over Marker (MOM) model to include a structured prior probability that accounts for the transcript location. The new technique exploits the fact that genetic markers are more likely to influence transcripts that share the same location on the genome. The third paper improves the analysis of Chromatin (Ch)-Immunoprecipitation (IP) (ChIP) microarray data. ChIP-chip data analysis estimates the motif of specific Transcription Factor Binding Sites (TFBSs) by comparing the IP DNA sample that is enriched for the TFBS and a control sample of general genomic DNA. The probes on the ChIP-chip array are uniformly spaced on the genome, and the probes that have relatively high intensity in the IP sample will have corresponding sequences that are likely to contain the TFBS motif. Present analytical methods use the array data to discover peaks or regions of IP enrichment then analyze the sequences of these peaks in a separate procedure to discover the motif. The proposed model will integrate enrichment peak finding and motif discovery through a Hidden Markov Model (HMM). Performance comparisons are made between the proposed HMM and the previously developed methods