Sequential Decision-Making for Drug Design: Towards closed-loop drug design

Drug design is a process of trial and error to design molecules with a desired response toward a biological target, with the ultimate goal of finding a new medication. It is estimated to be up to 10^{60} molecules that are of potential interest as drugs, making it a difficult problem to find suitable molecules. A crucial part of drug design is to design and determine what molecules should be experimentally tested, to determine their activity toward the biological target. To experimentally test the properties of a molecule, it has to be successfully made, often requiring a sequence of reactions to obtain the desired product. Machine learning can be utilized to predict the outcome of a reaction, helping to find successful reactions, but requires data for the reaction type of interest. This thesis presents a work that combinatorially investigates the use of active learning to acquire training data for reaching a certain level of predictive ability in predicting whether a reaction is successful or not. However, only a limited number of molecules can often be synthesized every time. Therefore, another line of work in this thesis investigates which designed molecules should be experimentally tested, given a budget of experiments, to sequentially acquire new knowledge. This is formulated as a multi-armed bandit problem and we propose an algorithm to solve this problem. To suggest potential drug molecules to choose from, recent advances in machine learning have also enabled the use of generative models to design novel molecules with certain predicted properties. Previous work has formulated this as a reinforcement learning problem with success in designing and optimizing molecules with drug-like properties. This thesis presents a systematic comparison of different reinforcement learning algorithms for string-based generation of drug molecules. This includes a study of different ways of learning from previous and current batches of samples during the iterative generation

Towards Thompson Sampling for Complex Bayesian Reasoning

Paper III, IV, and VI are not available as a part of the dissertation due to the copyright.Thompson Sampling (TS) is a state-of-art algorithm for bandit problems set in a Bayesian framework. Both the theoretical foundation and the empirical efficiency of TS is wellexplored for plain bandit problems. However, the Bayesian underpinning of TS means that TS could potentially be applied to other, more complex, problems as well, beyond the bandit problem, if suitable Bayesian structures can be found. The objective of this thesis is the development and analysis of TS-based schemes for more complex optimization problems, founded on Bayesian reasoning. We address several complex optimization problems where the previous state-of-art relies on a relatively myopic perspective on the problem. These includes stochastic searching on the line, the Goore game, the knapsack problem, travel time estimation, and equipartitioning. Instead of employing Bayesian reasoning to obtain a solution, they rely on carefully engineered rules. In all brevity, we recast each of these optimization problems in a Bayesian framework, introducing dedicated TS based solution schemes. For all of the addressed problems, the results show that besides being more effective, the TS based approaches we introduce are also capable of solving more adverse versions of the problems, such as dealing with stochastic liars.publishedVersio

Bandits on graphs and structures

We investigate the structural properties of certain sequential decision-making problems with limited feedback (bandits) in order to bring the known algorithmic solutions closer to a practical use. In the first part, we put a special emphasis on structures that can be represented as graphs on actions, in the second part we study the large action spaces that can be of exponential size in the number of base actions or even infinite. We show how to take advantage of structures over the actions and (provably) learn faster

A Non-Stochastic Learning Approach to Energy Efficient Mobility Management

Energy efficient mobility management is an important problem in modern wireless networks with heterogeneous cell sizes and increased nodes densities. We show that optimization-based mobility protocols cannot achieve long-Term optimal energy consumption, particularly for ultra-dense networks (UDNs). To address the complex dynamics of UDN, we propose a non-stochastic online-learning approach, which does not make any assumption on the statistical behavior of the small base station (SBS) activities. In addition, we introduce handover cost to the overall energy consumption, which forces the resulting solution to explicitly minimize frequent handovers. The proposed batched randomization with exponential weighting (BREW) algorithm relies on batching to explore in bulk, and hence reduces unnecessary handovers. We prove that the regret of BREW is sublinear in time, thus guaranteeing its convergence to the optimal SBS selection. We further study the robustness of the BREW algorithm to delayed or missing feedback. Moreover, we study the setting where SBSs can be dynamically turned ON and OFF. We prove that sublinear regret is impossible with respect to arbitrary SBS ON/OFF, and then develop a novel learning strategy, called ranking expert (RE), that simultaneously takes into account the handover cost and the availability of SBS. To address the high complexity of RE, we propose a contextual ranking expert (CRE) algorithm that only assigns experts in a given context. Rigorous regret bounds are proved for both RE and CRE with respect to the best expert. Simulations show that not only do the proposed mobility algorithms greatly reduce the system energy consumption, but they are also robust to various dynamics which are common in practical ultra-dense wireless networks. © 1983-2012 IEEE

Locating People of Interest in Social Networks

By representing relationships between social entities as a network, researchers can analyze them using a variety of powerful techniques. One key problem in social network analysis literature is identifying certain individuals (key players, most influential nodes) in a network. We consider the same problem in this dissertation, with the constraint that the individuals we are interested in identifying (People of Interest) are not necessarily the most important nodes in terms of the network structure. We propose an algorithm to find POIs, algorithms to collect data to find POIs, a framework to model POI behavior and an algorithm to predict POIs with guaranteed error rates. First, we propose a multi-objective optimization algorithm to find individuals who are expected to become stars in the future (rising stars), considering dynamic network data and multiple data types. Our algorithm outperforms the state of the art algorithm to find rising stars in academic data. Second, we propose two algorithms to collect data in a network crawling setting to locate POIs in dark networks. We consider potential errors that adversarial POIs can introduce to data collection process to hinder the analysis. We test and present our results on several real-world networks, and show that the proposed algorithms achieve up to a 340% improvement over the next best strategy. Next,We introduce the Adversarial Social Network Analysis game framework to model adversarial behavior of POIs towards a data collector in social networks. We run behavior experiments in Amazon Mechanical Turk and demonstrate the validity of the framework to study adversarial behavior by showing, 1) Participants understand their role, 2) Participants understand their objective in a game and, 3) Participants act as members of the adversarial group. Last, we show that node classification algorithms can be used to predict POIs in social networks. We then demonstrate how to utilize conformal prediction framework [103] to obtain guaranteed error bounds in POI prediction. Experimental results show that the Conformal Prediction framework can provide up to a 30% improvement in node classification algorithm accuracy while maintaining guaranteed error bounds on predictions

Racism in Modern Russia

In October 2013, one of the largest anti-migrant riots took place in Moscow. Clashes and arrests continued late into the night. Some in the crowd, which grew to several thousand people, could be heard chanting “Russia for the Russians” with their animus directed towards dark-skinned labor migrants from the southern border. The slogan “Russia for the Russians” is not a recent invention. It first gained notoriety in the very last years of the tsarist regime, appealing primarily to individuals drawn to the radical right. Analyzing a wide range of printed and visual sources, Racism in Modern Russia marks the first serious attempt to understand the history of racism over a span of 150 years. A brilliant examination of the complexities of racism, Eugene M. Avrutin’s panoramic book asks powerful questions about inequality and privilege, denigration and belonging, power and policy, and the complex historical links between race, whiteness, and geography. The ebook editions of this book are available open access under a CC BY-NC-ND 4.0 license on www.bloomsburycollections.com

The protection and security of vulnerable populations in complex emergencies using the Dadaab refugee camps in the north eastern province of Kenya as a case study

The past two decades has seen a dramatic upheaval in the international world order: the end of the Cold War, the 9/11 attacks and the subsequent 'War on Terror', increased Jihadist activities, the accelerated pace of globalization, climate change and the 2008 global financial crisis have contributed to fear, uncertainty, poverty, conflict, massive displacements of populations of asylum seekers and refugees globally and a proliferation of Protracted Refugee Situations (PRS), defined as situations in which refugees have been in exile 'for 5 years or more after their initial displacement, without immediate prospects for implementation of durable solutions. In the past two decades there has been a huge proliferation of these with more than 7.2 million refugees now trapped in these PRS, with a further 16 million internally displaced persons (IDPs) trapped in camps within their own countries. The Dadaab refugee complex in Kenya, which of as March 2012, holds over 463,000 refugees, is the most significant and extreme example in recent times of a PRS. It was established in 1991 following the collapse of the Somali Government of Dictator Siad Barre, and the disintegration of Somalia into the chaos that still exists today. PRS such as Dadaab raise particular issues about humanitarianism in terms of aid, protection, security, human rights and the actions (or inaction) of the various stakeholders on an international, national and local level. This thesis investigates these issues by the use of a case study methodology on Dadaab as a PRS, framed in the context of humanitarianism and in particular the issues that arise in terms of how the international community, the UN system and individual states provide assistance and protection to vulnerable populations. Although the refugee camps have been in existence (as of 2012) for over 20 years, there has never been such a detailed study of Dadaab (or any other PRS) undertaken to date and would be of interest to academics in the areas of international relations, refugee/migration studies and global Governance as well as practitioners in both humanitarian response and developmen