Learning Bayesian networks based on optimization approaches

Learning accurate classifiers from preclassified data is a very active research topic in machine learning and artifcial intelligence. There are numerous classifier paradigms, among which Bayesian Networks are very effective and well known in domains with uncertainty. Bayesian Networks are widely used representation frameworks for reasoning with probabilistic information. These models use graphs to capture dependence and independence relationships between feature variables, allowing a concise representation of the knowledge as well as efficient graph based query processing algorithms. This representation is defined by two components: structure learning and parameter learning. The structure of this model represents a directed acyclic graph. The nodes in the graph correspond to the feature variables in the domain, and the arcs (edges) show the causal relationships between feature variables. A directed edge relates the variables so that the variable corresponding to the terminal node (child) will be conditioned on the variable corresponding to the initial node (parent). The parameter learning represents probabilities and conditional probabilities based on prior information or past experience. The set of probabilities are represented in the conditional probability table. Once the network structure is constructed, the probabilistic inferences are readily calculated, and can be performed to predict the outcome of some variables based on the observations of others. However, the problem of structure learning is a complex problem since the number of candidate structures grows exponentially when the number of feature variables increases. This thesis is devoted to the development of learning structures and parameters in Bayesian Networks. Different models based on optimization techniques are introduced to construct an optimal structure of a Bayesian Network. These models also consider the improvement of the Naive Bayes' structure by developing new algorithms to alleviate the independence assumptions. We present various models to learn parameters of Bayesian Networks; in particular we propose optimization models for the Naive Bayes and the Tree Augmented Naive Bayes by considering different objective functions. To solve corresponding optimization problems in Bayesian Networks, we develop new optimization algorithms. Local optimization methods are introduced based on the combination of the gradient and Newton methods. It is proved that the proposed methods are globally convergent and have superlinear convergence rates. As a global search we use the global optimization method, AGOP, implemented in the open software library GANSO. We apply the proposed local methods in the combination with AGOP. Therefore, the main contributions of this thesis include (a) new algorithms for learning an optimal structure of a Bayesian Network; (b) new models for learning the parameters of Bayesian Networks with the given structures; and finally (c) new optimization algorithms for optimizing the proposed models in (a) and (b). To validate the proposed methods, we conduct experiments across a number of real world problems. Print version is available at: http://library.federation.edu.au/record=b1804607~S4Doctor of Philosoph

Incorporation of relational information in feature representation for online handwriting recognition of Arabic characters

Interest in online handwriting recognition is increasing due to market demand for both improved performance and for extended supporting scripts for digital devices. Robust handwriting recognition of complex patterns of arbitrary scale, orientation and location is elusive to date because reaching a target recognition rate is not trivial for most of the applications in this field. Cursive scripts such as Arabic and Persian with complex character shapes make the recognition task even more difficult. Challenges in the discrimination capability of handwriting recognition systems depend heavily on the effectiveness of the features used to represent the data, the types of classifiers deployed and inclusive databases used for learning and recognition which cover variations in writing styles that introduce natural deformations in character shapes. This thesis aims to improve the efficiency of online recognition systems for Persian and Arabic characters by presenting new formal feature representations, algorithms, and a comprehensive database for online Arabic characters. The thesis contains the development of the first public collection of online handwritten data for the Arabic complete-shape character set. New ideas for incorporating relational information in a feature representation for this type of data are presented. The proposed techniques are computationally efficient and provide compact, yet representative, feature vectors. For the first time, a hybrid classifier is used for recognition of online Arabic complete-shape characters based on the idea of decomposing the input data into variables representing factors of the complete-shape characters and the combined use of the Bayesian network inference and support vector machines. We advocate the usefulness and practicality of the features and recognition methods with respect to the recognition of conventional metrics, such as accuracy and timeliness, as well as unconventional metrics. In particular, we evaluate a feature representation for different character class instances by its level of separation in the feature space. Our evaluation results for the available databases and for our own database of the characters' main shapes confirm a higher efficiency than previously reported techniques with respect to all metrics analyzed. For the complete-shape characters, our techniques resulted in a unique recognition efficiency comparable with the state-of-the-art results for main shape characters

Sparse graphical models for cancer signalling

Protein signalling networks play a key role in cellular function, and their dysregulation is central to many diseases, including cancer. Recent advances in biochemical technology have begun to allow high-throughput, data-driven studies of signalling. In this thesis, we investigate multivariate statistical methods, rooted in sparse graphical models, aimed at probing questions in cancer signalling. First, we propose a Bayesian variable selection method for identifying subsets of proteins that jointly in uence an output of interest, such as drug response. Ancillary biological information is incorporated into inference using informative prior distributions. Prior information is selected and weighted in an automated manner using an empirical Bayes formulation. We present examples of informative pathway and network-based priors, and illustrate the proposed method on both synthetic and drug response data. Second, we use dynamic Bayesian networks to perform structure learning of context-specific signalling network topology from proteomic time-course data. We exploit a connection between variable selection and network structure learning to efficiently carry out exact inference. Existing biology is incorporated using informative network priors, weighted automatically by an empirical Bayes approach. The overall approach is computationally efficient and essentially free of user-set parameters. We show results from an empirical investigation, comparing the approach to several existing methods, and from an application to breast cancer cell line data. Hypotheses are generated regarding novel signalling links, some of which are validated by independent experiments. Third, we describe a network-based clustering approach for the discovery of cancer subtypes that differ in terms of subtype-specific signalling network structure. Model-based clustering is combined with penalised likelihood estimation of undirected graphical models to allow simultaneous learning of cluster assignments and cluster-specific network structure. Results are shown from an empirical investigation comparing several penalisation regimes, and an application to breast cancer proteomic data

Asymptotic bounds to outputs of the exact weak solution of the three-dimensional Helmholtz equation

In engineering practice, the design is based on certain design quantities or "outputs of interest" which are functionals of field variables such as displacement, velocity field, or pressure. In order to gain confidence in the numerical approximation of "outputs," a method of obtaining sharp, rigorous upper and lower bounds to outputs of the exact solution have been developed for symmetric and coercive problems (the Poisson equation and the elasticity equation), for non-symmetric coercive problems (advection-diffusion-reaction equation), and more recently for certain constrained problems (Stokes equation). In this thesis we develop the method for the Helmholtz equation. The common approach relies on decomposing the global problem into independent local elemental sub-problems by relaxing the continuity along the edges of a partitioning of the entire domain, using approximate Lagrange multipliers. The method exploits the Lagrangian saddle point property by recasting the output problem as a constrained minimization problem. The upper and lower computed bounds then hold for all levels of refinement and are shown to approach the exact solution at the same rate as its underlying finite element approach. The certificate of precision can then determine the best as well as the worst case scenario in an engineering design problem. This thesis addresses bounds to outputs of interest for the complex Helmholtz equation. The Helmholtz equation is in general non-coercive for high wave numbers and therefore, the previous approaches that relied on duality theory of convex minimization do not directly apply. Only in the asymptotic regime does the Helmholtz equation become coercive, and reliable (guaranteed) bounds can thus be obtained. Therefore, in order to achieve good bounds, several new ingredients have been introduced. The bounds procedure is firstly formulated with appropriate extension to complex-valued equations. Secondly, in the computation of the inter-subdomain continuity multipliers we follow the FETI-H approach and regularize the system matrix with a complex term to make the system non-singular. Finally, in order to obtain sharper output bounds in the presence of pollution errors, higher order nodal spectral element method is employed which has several computational advantages over the traditional finite element approach. We performed verification of our results and demonstrate the bounding properties for the Helmholtz problem

Sparse graphical models for cancer signalling

Protein signalling networks play a key role in cellular function, and their dysregulation is central to many diseases, including cancer. Recent advances in biochemical technology have begun to allow high-throughput, data-driven studies of signalling. In this thesis, we investigate multivariate statistical methods, rooted in sparse graphical models, aimed at probing questions in cancer signalling. First, we propose a Bayesian variable selection method for identifying subsets of proteins that jointly in uence an output of interest, such as drug response. Ancillary biological information is incorporated into inference using informative prior distributions. Prior information is selected and weighted in an automated manner using an empirical Bayes formulation. We present examples of informative pathwayand network-based priors, and illustrate the proposed method on both synthetic and drug response data. Second, we use dynamic Bayesian networks to perform structure learning of context-specific signalling network topology from proteomic time-course data. We exploit a connection between variable selection and network structure learning to efficiently carry out exact inference. Existing biology is incorporated using informative network priors, weighted automatically by an empirical Bayes approach. The overall approach is computationally efficient and essentially free of user-set parameters. We show results from an empirical investigation, comparing the approach to several existing methods, and from an application to breast cancer cell line data. Hypotheses are generated regarding novel signalling links, some of which are validated by independent experiments. Third, we describe a network-based clustering approach for the discovery of cancer subtypes that differ in terms of subtype-specific signalling network structure. Model-based clustering is combined with penalised likelihood estimation of undirected graphical models to allow simultaneous learning of cluster assignments and cluster-specific network structure. Results are shown from an empirical investigation comparing several penalisation regimes, and an application to breast cancer proteomic data.EThOS - Electronic Theses Online ServiceEngineering and Physical Sciences Research Council (EPSRC)GBUnited Kingdo

Audio-coupled video content understanding of unconstrained video sequences

Unconstrained video understanding is a difficult task. The main aim of this thesis is to recognise the nature of objects, activities and environment in a given video clip using both audio and video information. Traditionally, audio and video information has not been applied together for solving such complex task, and for the first time we propose, develop, implement and test a new framework of multi-modal (audio and video) data analysis for context understanding and labelling of unconstrained videos. The framework relies on feature selection techniques and introduces a novel algorithm (PCFS) that is faster than the well-established SFFS algorithm. We use the framework for studying the benefits of combining audio and video information in a number of different problems. We begin by developing two independent content recognition modules. The first one is based on image sequence analysis alone, and uses a range of colour, shape, texture and statistical features from image regions with a trained classifier to recognise the identity of objects, activities and environment present. The second module uses audio information only, and recognises activities and environment. Both of these approaches are preceded by detailed pre-processing to ensure that correct video segments containing both audio and video content are present, and that the developed system can be made robust to changes in camera movement, illumination, random object behaviour etc. For both audio and video analysis, we use a hierarchical approach of multi-stage classification such that difficult classification tasks can be decomposed into simpler and smaller tasks. When combining both modalities, we compare fusion techniques at different levels of integration and propose a novel algorithm that combines advantages of both feature and decision-level fusion. The analysis is evaluated on a large amount of test data comprising unconstrained videos collected for this work. We finally, propose a decision correction algorithm which shows that further steps towards combining multi-modal classification information effectively with semantic knowledge generates the best possible results

Scalable Analysis, Verification and Design of IC Power Delivery

Due to recent aggressive process scaling into the nanometer regime, power delivery network design faces many challenges that set more stringent and specific requirements to the EDA tools. For example, from the perspective of analysis, simulation efficiency for large grids must be improved and the entire network with off-chip models and nonlinear devices should be able to be analyzed. Gated power delivery networks have multiple on/off operating conditions that need to be fully verified against the design requirements. Good power delivery network designs not only have to save the wiring resources for signal routing, but also need to have the optimal parameters assigned to various system components such as decaps, voltage regulators and converters. This dissertation presents new methodologies to address these challenging problems. At first, a novel parallel partitioning-based approach which provides a flexible network partitioning scheme using locality is proposed for power grid static analysis. In addition, a fast CPU-GPU combined analysis engine that adopts a boundary-relaxation method to encompass several simulation strategies is developed to simulate power delivery networks with off-chip models and active circuits. These two proposed analysis approaches can achieve scalable simulation runtime. Then, for gated power delivery networks, the challenge brought by the large verification space is addressed by developing a strategy that efficiently identifies a number of candidates for the worst-case operating condition. The computation complexity is reduced from O(2^N) to O(N). At last, motivated by a proposed two-level hierarchical optimization, this dissertation presents a novel locality-driven partitioning scheme to facilitate divide-and-conquer-based scalable wire sizing for large power delivery networks. Simultaneous sizing of multiple partitions is allowed which leads to substantial runtime improvement. Moreover, the electric interactions between active regulators/converters and passive networks and their influences on key system design specifications are analyzed comprehensively. With the derived design insights, the system-level co-design of a complete power delivery network is facilitated by an automatic optimization flow. Results show significant performance enhancement brought by the co-design

Protein-protein interactions: impact of solvent and effects of fluorination

Proteins have an indispensable role in the cell. They carry out a wide variety of structural, catalytic and signaling functions in all known biological systems. To perform their biological functions, proteins establish interactions with other bioorganic molecules including other proteins. Therefore, protein-protein interactions is one of the central topics in molecular biology. My thesis is devoted to three different topics in the field of protein-protein interactions. The first one focuses on solvent contribution to protein interfaces as it is an important component of protein complexes. The second topic discloses the structural and functional potential of fluorine's unique properties, which are attractive for protein design and engineering not feasible within the scope of canonical amino acids. The last part of this thesis is a study of the impact of charged amino acid residues within the hydrophobic interface of a coiled-coil system, which is one of the well-established model systems for protein-protein interactions studies. I. The majority of proteins interact in vivo in solution, thus studies of solvent impact on protein-protein interactions could be crucial for understanding many processes in the cell. However, though solvent is known to be very important for protein-protein interactions in terms of structure, dynamics and energetics, its effects are often disregarded in computational studies because a detailed solvent description requires complex and computationally demanding approaches. As a consequence, many protein residues, which establish water-mediated interactions, are neither considered in an interface definition. In the previous work carried out in our group the protein interfaces database (SCOWLP) has been developed. This database takes into account interfacial solvent and based on this classifies all interfacial protein residues of the PDB into three classes based on their interacting properties: dry (direct interaction), dual (direct and water-mediated interactions), and wet spots (residues interacting only through one water molecule). To define an interaction SCOWLP considers a donor–acceptor distance for hydrogen bonds of 3.2 Å, for salt bridges of 4 Å, and for van der Waals contacts the sum of the van der Waals radii of the interacting atoms. In previous studies of the group, statistical analysis of a non-redundant protein structure dataset showed that 40.1% of the interfacial residues participate in water-mediated interactions, and that 14.5% of the total residues in interfaces are wet spots. Moreover, wet spots have been shown to display similar characteristics to residues contacting water molecules in cores or cavities of proteins. The goals of this part of the thesis were: 1. to characterize the impact of solvent in protein-protein interactions 2. to elucidate possible effects of solvent inclusion into the correlated mutations approach for protein contacts prediction To study solvent impact on protein interfaces a molecular dynamics (MD) approach has been used. This part of the work is elaborated in section 2.1 of this thesis. We have characterized properties of water-mediated protein interactions at residue and solvent level. For this purpose, an MD analysis of 17 representative complexes from SH3 and immunoglobulin protein families has been performed. We have shown that the interfacial residues interacting through a single water molecule (wet spots) are energetically and dynamically very similar to other interfacial residues. At the same time, water molecules mediating protein interactions have been found to be significantly less mobile than surface solvent in terms of residence time. Calculated free energies indicate that these water molecules should significantly affect formation and stability of a protein-protein complex. The results obtained in this part of the work also suggest that water molecules in protein interfaces contribute to the conservation of protein interactions by allowing more sequence variability in the interacting partners, which has important implications for the use of the correlated mutations concept in protein interactions studies. This concept is based on the assumption that interacting protein residues co-evolve, so that a mutation in one of the interacting counterparts is compensated by a mutation in the other. The study presented in section 2.2 has been carried out to prove that an explicit introduction of solvent into the correlated mutations concept indeed yields qualitative improvement of existing approaches. For this, we have used the data on interfacial solvent obtained from the SCOWLP database (the whole PDB) to construct a “wet” similarity matrix. This matrix has been used for prediction of protein contacts together with a well-established “dry” matrix. We have analyzed two datasets containing 50 domains and 10 domain pairs, and have compared the results obtained by using several combinations of both “dry” and “wet” matrices. We have found that for predictions for both intra- and interdomain contacts the introduction of a combination of a “dry” and a “wet” similarity matrix improves the predictions in comparison to the “dry” one alone. Our analysis opens up the idea that the consideration of water may have an impact on the improvement of the contact predictions obtained by correlated mutations approaches. There are two principally novel aspects in this study in the context of the used correlated mutations methodology : i) the first introduction of solvent explicitly into the correlated mutations approach; ii) the use of the definition of protein-protein interfaces, which is essentially different from many other works in the field because of taking into account physico-chemical properties of amino acids and not being exclusively based on distance cut-offs. II. The second part of the thesis is focused on properties of fluorinated amino acids in protein environments. In general, non-canonical amino acids with newly designed side-chain functionalities are powerful tools that can be used to improve structural, catalytic, kinetic and thermodynamic properties of peptides and proteins, which otherwise are not feasible within the use of canonical amino acids. In this context fluorinated amino acids have increasingly gained in importance in protein chemistry because of fluorine's unique properties: high electronegativity and a small atomic size. Despite the wide use of fluorine in drug design, properties of fluorine in protein environments have not been yet extensively studied. The aims of this part of the dissertation were: 1. to analyze the basic properties of fluorinated amino acids such as electrostatic and geometric characteristics, hydrogen bonding abilities, hydration properties and conformational preferences (section 3.1) 2. to describe the behavior of fluorinated amino acids in systems emulating protein environments (section 3.2, section 3.3) First, to characterize fluorinated amino acids side chains we have used fluorinated ethane derivatives as their simplified models and applied a quantum mechanics approach. Properties such as charge distribution, dipole moments, volumes and size of the fluoromethylated groups within the model have been characterized. Hydrogen bonding properties of these groups have been compared with the groups typically presented in natural protein environments. We have shown that hydrogen and fluorine atoms within these fluoromethylated groups are weak hydrogen bond donors and acceptors. Nevertheless they should not be disregarded for applications in protein engineering. Then, we have implemented four fluorinated L-amino acids for the AMBER force field and characterized their conformational and hydration properties at the MD level. We have found that hydrophobicity of fluorinated side chains grows with the number of fluorine atoms and could be explained in terms of high electronegativity of fluorine atoms and spacial demand of fluorinated side-chains. These data on hydration agrees with the results obtained in the experimental work performed by our collaborators. We have rationally engineered systems that allow us to study fluorine properties and extract results that could be extrapolated to proteins. For this, we have emulated protein environments by introducing fluorinated amino acids into a parallel coiled-coil and enzyme-ligand chymotrypsin systems. The results on fluorination effect on coiled-coil dimerization and substrate affinities in the chymotrypsin active site obtained by MD, molecular docking and free energy calculations are in strong agreement with experimental data obtained by our collaborators. In particular, we have shown that fluorine content and position of fluorination can considerably change the polarity and steric properties of an amino acid side chain and, thus, can influence the properties that a fluorinated amino acid reveals within a native protein environment. III. Coiled-coils typically consist of two to five right-handed α-helices that wrap around each other to form a left-handed superhelix. The interface of two α-helices is usually represented by hydrophobic residues. However, the analysis of protein databases revealed that in natural occurring proteins up to 20% of these positions are populated by polar and charged residues. The impact of these residues on stability of coiled-coil system is not clear. MD simulations together with free energy calculations have been utilized to estimate favourable interaction partners for uncommon amino acids within the hydrophobic core of coiled-coils (Chapter 4). Based on these data, the best hits among binding partners for one strand of a coiled-coil bearing a charged amino acid in a central hydrophobic core position have been selected. Computational data have been in agreement with the results obtained by our collaborators, who applied phage display technology and CD spectroscopy. This combination of theoretical and experimental approaches allowed to get a deeper insight into the stability of the coiled-coil system. To conclude, this thesis widens existing concepts of protein structural biology in three areas of its current importance. We expand on the role of solvent in protein interfaces, which contributes to the knowledge of physico-chemical properties underlying protein-protein interactions. We develop a deeper insight into the understanding of the fluorine's impact upon its introduction into protein environments, which may assist in exploiting the full potential of fluorine's unique properties for applications in the field of protein engineering and drug design. Finally we investigate the mechanisms underlying coiled-coil system folding. The results presented in the thesis are of definite importance for possible applications (e.g. introduction of solvent explicitly into the scoring function) into protein folding, docking and rational design methods. The dissertation consists of four chapters: ● Chapter 1 contains an introduction to the topic of protein-protein interactions including basic concepts and an overview of the present state of research in the field. ● Chapter 2 focuses on the studies of the role of solvent in protein interfaces. ● Chapter 3 is devoted to the work on fluorinated amino acids in protein environments. ● Chapter 4 describes the study of coiled-coils folding properties. The experimental parts presented in Chapters 3 and 4 of this thesis have been performed by our collaborators at FU Berlin. Sections 2.1, 2.2, 3.1, 3.2 and Chapter 4 have been submitted/published in peer-reviewed international journals. Their organization follows a standard research article structure: Abstract, Introduction, Methodology, Results and discussion, and Conclusions. Section 3.3, though not published yet, is also organized in the same way. The literature references are summed up together at the end of the thesis to avoid redundancy within different chapters