Regularizing Deep Networks by Modeling and Predicting Label Structure

We construct custom regularization functions for use in supervised training of deep neural networks. Our technique is applicable when the ground-truth labels themselves exhibit internal structure; we derive a regularizer by learning an autoencoder over the set of annotations. Training thereby becomes a two-phase procedure. The first phase models labels with an autoencoder. The second phase trains the actual network of interest by attaching an auxiliary branch that must predict output via a hidden layer of the autoencoder. After training, we discard this auxiliary branch. We experiment in the context of semantic segmentation, demonstrating this regularization strategy leads to consistent accuracy boosts over baselines, both when training from scratch, or in combination with ImageNet pretraining. Gains are also consistent over different choices of convolutional network architecture. As our regularizer is discarded after training, our method has zero cost at test time; the performance improvements are essentially free. We are simply able to learn better network weights by building an abstract model of the label space, and then training the network to understand this abstraction alongside the original task.Comment: to appear at CVPR 201

Performance of machine-learning scoring functions in structure-based virtual screening

Classical scoring functions have reached a plateau in their performance in virtual screening and binding affinity prediction. Recently, machine-learning scoring functions trained on protein-ligand complexes have shown great promise in small tailored studies. They have also raised controversy, specifically concerning model overfitting and applicability to novel targets. Here we provide a new ready-to-use scoring function (RF-Score-VS) trained on 15 426 active and 893 897 inactive molecules docked to a set of 102 targets. We use the full DUD-E data sets along with three docking tools, five classical and three machine-learning scoring functions for model building and performance assessment. Our results show RF-Score-VS can substantially improve virtual screening performance: RF-Score-VS top 1% provides 55.6% hit rate, whereas that of Vina only 16.2% (for smaller percent the difference is even more encouraging: RF-Score-VS top 0.1% achieves 88.6% hit rate for 27.5% using Vina). In addition, RF-Score-VS provides much better prediction of measured binding affinity than Vina (Pearson correlation of 0.56 and -0.18, respectively). Lastly, we test RF-Score-VS on an independent test set from the DEKOIS benchmark and observed comparable results. We provide full data sets to facilitate further research in this area (http://github.com/oddt/rfscorevs) as well as ready-to-use RF-Score-VS (http://github.com/oddt/rfscorevs_binary)

An Analysis of the Genetic Algorithm and Abstract Search Space Visualisation

The Genetic Algorithm (Holland, 1975) is a powerful search technique based upon the principles of Darwinian evolution. In its simplest form the GA consists of three main operators - crossover, mutation and selection. The principal theoretical treatment of the Genetic Algorithm (GA) is provided by the Schema Theorem and building block hypothesis (Holland, 1975). The building block hypothesis describes the GA search process as the combination, sampling and recombination of fragments of solutions known as building blocks. The crossover operator is responsible for the combination of building blocks, whilst the selection operator allocates increasing numbers of samples to good building blocks. Thus the GA constructs the optimal (or near-optimal) solution from those fragments of solutions which are, in some sense, optimal. The first part of this thesis documents the development of a technique for the isolation of building blocks from the populations of the GA. This technique is shown to extract exactly those building blocks of interest - those which are sampled most regularly by the GA. These building blocks are used to empirically investigate the validity of the building block hypothesis. It is shown that good building blocks do not combine to form significantly better solution fragments than those resulting from the addition of randomly generated building blocks to good building blocks. This results casts some doubt onto the value of the building block hypothesis as an account of the GA search process (at least for the functions used during these experiments). The second part of this thesis describes an alternative account of the action of crossover. This account is an approximation of the geometric effect of crossover upon the population of samples maintained by the GA. It is shown that, for a simple function, this description of the crossover operator is sufficiently accurate to warrant further investigation. A pair of performance models for the GA upon this function are derived and shown to be accurate for a wide range of crossover schemes. Finally, the GA search process is described in terms of this account of the crossover operator and parallels are drawn with the search process of the simulated annealing algorithm (Kirkpatrick et al, 1983). The third and final part of this thesis describes a technique for the visualisation of high dimensional surfaces, such as are defined by functions of many parameters. This technique is compared to the statistical technique of projection pursuit regression (Friedman & Tukey, 1974) and is shown to compare favourably both in terms of computational expense and quantitative accuracy upon a wide range of test functions. A fundamental flaw of this technique is that it may produce poor visualisations when applied to functions with a small high frequency (or order) components

GRACE at ONE-LOOP: Automatic calculation of 1-loop diagrams in the electroweak theory with gauge parameter independence checks

We describe the main building blocks of a generic automated package for the calculation of Feynman diagrams. These blocks include the generation and creation of a model file, the graph generation, the symbolic calculation at an intermediate level of the Dirac and tensor algebra, implementation of the loop integrals, the generation of the matrix elements or helicity amplitudes, methods for the phase space integrations and eventually the event generation. The report focuses on the fully automated systems for the calculation of physical processes based on the experience in developing GRACE-loop. As such, a detailed description of the renormalisation procedure in the Standard Model is given emphasizing the central role played by the non-linear gauge fixing conditions for the construction of such automated codes. The need for such gauges is better appreciated when it comes to devising efficient and powerful algorithms for the reduction of the tensorial structures of the loop integrals. A new technique for these reduction algorithms is described. Explicit formulae for all two-point functions in a generalised non-linear gauge are given, together with the complete set of counterterms. We also show how infrared divergences are dealt with in the system. We give a comprehensive presentation of some systematic test-runs which have been performed at the one-loop level for a wide variety of two-to-two processes to show the validity of the gauge check. These cover fermion-fermion scattering, gauge boson scattering into fermions, gauge bosons and Higgs bosons scattering processes. Comparisons with existing results on some one-loop computation in the Standard Model show excellent agreement. We also briefly recount some recent development concerning the calculation of mutli-leg one-loop corrections.Comment: 131 pages. Manuscript expanded quite substantially with the inclusion of an overview of automatic systems for the calculation of Feynman diagrams both at tree-level and one-loop. Other additions include issues of regularisation, width effects and renormalisation with unstable particles and reduction of 5- and 6-point functions. This is a preprint version, final version to appear as a Phys. Re

Enhancing High-dimensional Bayesian Optimization by Optimizing the Acquisition Function Maximizer Initialization

Bayesian optimization (BO) is widely used to optimize black-box functions. It works by first building a surrogate for the objective and quantifying the uncertainty in that surrogate. It then decides where to sample by maximizing an acquisition function defined by the surrogate model. Prior approaches typically use randomly generated raw samples to initialize the acquisition function maximizer. However, this strategy is ill-suited for high-dimensional BO. Given the large regions of high posterior uncertainty in high dimensions, a randomly initialized acquisition function maximizer is likely to focus on areas with high posterior uncertainty, leading to overly exploring areas that offer little gain. This paper provides the first comprehensive empirical study to reveal the importance of the initialization phase of acquisition function maximization. It proposes a better initialization approach by employing multiple heuristic optimizers to leverage the knowledge of already evaluated samples to generate initial points to be explored by an acquisition function maximizer. We evaluate our approach on widely used synthetic test functions and real-world applications. Experimental results show that our techniques, while simple, can significantly enhance the standard BO and outperforms state-of-the-art high-dimensional BO techniques by a large margin in most test cases

An Empirical Examination of the Impact of Item Parameters on IRT Information Functions in Mixed Format Tests

IRT, also referred as modern test theory , offers many advantages over CTT-based methods in test development. Specifically, an IRT information function has the capability to build a test that has the desired precision of measurement for any defined proficiency scale when a sufficient number of test items are available. This feature is extremely useful when the information is used for decision making, for instance, whether an examinee attain certain mastery level. Computerized adaptive testing (CAT) is one of the many examples using IRT information functions in test construction. The purposes of this study were as follows: (1) to examine the consequences of improving the test quality through the addition of more discriminating items with different item formats; (2) to examine the effect of having a test where its difficulty does not align with the ability level of the intended population; (3) to investigate the change in decision consistency and decision accuracy; and (4) to understand changes in expected information when test quality is either improved or degraded, using both empirical and simulated data. Main findings from the study were as follows: (1) increasing the discriminating power of any types of items generally increased the level of information; however, sometimes it could bring adverse effect to the extreme ends of the ability continuum; (2) it was important to have more items that were targeted at the population of interest, otherwise, no matter how good the quality of the items may be, they were of less value in test development when they were not targeted to the distribution of candidate ability or at the cutscores; (3) decision consistency (DC), Kappa statistic, and decision accuracy (DA) increased with better quality items; (4) DC and Kappa were negatively affected when difficulty of the test did not match with the ability of the intended population; however, the effect was less severe if the test was easier than needed; (5) tests with more better quality items lowered false positive (FP) and false negative (FN) rate at the cutscores; (6) when test difficulty did not match with the ability of the target examinees, in general, both FP and FN rates increased; (7) polytomous items tended to yield more information than dichotomously scored items, regardless of the discriminating parameter and difficulty of the item; and (8) the more score categories an item had, the more information it could provide. Findings from this thesis should help testing agencies and practitioners to have better understanding of the item parameters on item and test information functions. This understanding is crucial for the improvement of the item bank quality and ultimately on how to build better tests that could provide more accurate proficiency classifications. However, at the same time, item writers should be conscientious about the fact that the item information function is merely a statistical tool for building a good test, other criteria should also be considered, for example, content balancing and content validity

Utilizing machine learning algorithms in the ensemble-based optimization (EnOpt) ‎method for enhancing gradient estimation‎

High or even prohibitive computational cost is one of the key limitations of robust ‎optimization using the Ensemble-based Optimization (EnOpt) approach, especially when a ‎computationally demanding forward model is involved (e.g., a reservoir simulation model). ‎It is because, in EnOpt, many realizations of the forward model are considered to represent ‎uncertainty, and many runs of forward modeling need to be performed to estimate gradients ‎for optimization. This work aims to develop, investigate, and discuss an approach, named ‎EnOpt-ML in the thesis, of utilizing machine learning (ML) methods for speeding up ‎EnOpt, particularly for the gradient estimation in the EnOpt method.‎ The significance of any deviations is investigated on three different optimization test ‎functions: Himmelblau, Bukin function number 6 and Rosenbrock for their different ‎characteristics. A thousand simulations are performed for each configuration setting to do ‎the analyses, compare means and standard deviations of the ensembles. Singled out cases ‎are shown as examples of gradient learning curves differences between EnOpt and EnOpt-‎ML, and the spread of their samples over the test function.‎ Objectives:‎ Objective1: Building of a code with a main function that would allow easy configurations ‎and tweaking of parameters of EnOpt, Machine learning (ML) algorithms and test function ‎or objective functions in general (with two variables). Codes necessary for test functions, ‎ML algorithms, plotting and simulation data saving files are defined outside of that main ‎function.‎ The code is attached in the Appendix. ‎ Objective2: Testing and analysis of results to detect any special improvement with EnOpt-‎ML compared to EnOpt. The use of Himmelblau as a primary test function was with a ‎modification of specific parameters, one at a time, starting with a base configuration case ‎for possible comparisons. After gathering traits of effects of those configurations, an ‎example where the improvement could show interesting were presented and then applied to ‎the other two test functions and analyzed. ‎ The main objective then has been to reduce the number of times the objective function is ‎evaluated while not considerably reducing the optimization quality. ‎ EnOpt-ML yielded slightly better results when compared to EnOpt under the same ‎conditions when fixing a maximum objective function evaluations through the number of ‎samples and the iteration at which this number is reduced.

Statistical analysis, ciphertext only attack, improvement of generic quasigroup string transformation and dynamic string transformation

Algebraic functions are the primitives that strengthen the cryptographic algorithms to ensure confidentiality of data and information. There is need for continues development of new and improvement of existing primitives. Quasigroup String transformation is one of those primitives that have many applications in cryptographic algorithms, Hash functions, and Pseudo-Random Number Generators. It is obvious that randomness and unpredictability is the requirement of every Cryptographic primitive. Most of those string transformations have not been implemented properly neither do they have security analysis. Cryptanalysis of existing scheme is as important as building new ones. In this paper, generic Quasigroup sting transformation is analyzed and found vulnerable to Ciphertext-Only-Attack. An adversary can compute the ciphertext to get the plaintext without prior knowledge of the plaintext. Pseudorandom numbers produced with generic string transformation can be reversed back to the original input with little effort. Therefore the generic quasigroup string transformation is compared with recently introduced string transformation and it is expected to provide better randomness and resistant to ciphertext-only-Attack. The proposed string transformation is suitable to one-way functions such as Hash functions, and pseudorandom number generators to mitigate the vulnerability of quasigroup string transformation to Ciphertext-Only-Attack. While the dynamic string transformation increase the difficulty level of predicting the substitution table used. The algorithms will be compared in terms of randomness using NIST statistical test suit, correlation Assessment and frequency Distribution

The Effect of Spiritualism on the Cognitive Functions on Learning and Memory

Last summer I conducted a pilot study which researched whether better working memory would be documented among an experimental group (individuals who report being spiritual) as compared to the control group (individuals who report being non-spiritual).Total scores showed a significantly higher sense of spiritualism among the spiritual participants vs. the non spiritual participants (p \u3c .001) along with a significant improvement in working memory for spiritual participants vs. non spiritual participants (p = .027). The results of this study documented significantly better performance on a task measuring emotional learning and memory among individuals who reported being spiritual as opposed to individuals who reported being non-spiritual. These findings build on prior studies suggesting the effect of positive emotions on broadening cognitive processes (Strauss & Allen, 2003). My current study is building on what my prior findings have suggested and studies the effect of spiritualism on the cognitive functions of learning and memory. In addition to the Daily Spiritual Experiences Scale (DSES) used in the pilot study, I am including the Spirituality Index of Well-Being, the Wisconsin Quality of Life Questionnaire, and the Test of Memory and Learning-2 (TOMAL-2). At this time, the study is still ongoing