Reinforcement Learning for Machine Translation: from Simulations to Real-World Applications

If a machine translation is wrong, how we can tell the underlying model to fix it? Answering this question requires (1) a machine learning algorithm to define update rules, (2) an interface for feedback to be submitted, and (3) expertise on the side of the human who gives the feedback. This thesis investigates solutions for machine learning updates, the suitability of feedback interfaces, and the dependency on reliability and expertise for different types of feedback. We start with an interactive online learning scenario where a machine translation (MT) system receives bandit feedback (i.e. only once per source) instead of references for learning. Policy gradient algorithms for statistical and neural MT are developed to learn from absolute and pairwise judgments. Our experiments on domain adaptation with simulated online feedback show that the models can largely improve under weak feedback, with variance reduction techniques being very effective. In production environments offline learning is often preferred over online learning. We evaluate algorithms for counterfactual learning from human feedback in a study on eBay product title translations. Feedback is either collected via explicit star ratings from users, or implicitly from the user interaction with cross-lingual product search. Leveraging implicit feedback turns out to be more successful due to lower levels of noise. We compare the reliability and learnability of absolute Likert-scale ratings with pairwise preferences in a smaller user study, and find that absolute ratings are overall more effective for improvements in down-stream tasks. Furthermore, we discover that error markings provide a cheap and practical alternative to error corrections. In a generalized interactive learning framework we propose a self-regulation approach, where the learner, guided by a regulator module, decides which type of feedback to choose for each input. The regulator is reinforced to find a good trade-off between supervision effect and cost. In our experiments, it discovers strategies that are more efficient than active learning and standard fully supervised learning

Accurate and Robust Mechanical Modeling of Proteins

Through their motion, proteins perform essential functions in the living cell. Although we cannot observe protein motion directly, over 68,000 crystal structures are freely available from the Protein Data Bank. Computational protein rigidity analysis systems leverage this data, building a mechanical model from atoms and pairwise interactions determined from a static 3D structure. The rigid and flexible components of the model are then calculated with a pebble game algorithm, predicting a protein\u27s flexibility with much more computational efficiency than physical simulation. In prior work with rigidity analysis systems, the available modeling options were hard-coded, and evaluation was limited to case studies. The focus of this thesis is improving accuracy and robustness of rigidity analysis systems. The first contribution is in new approaches to mechanical modeling of noncovalent interactions, namely hydrogen bonds and hydrophobic interactions. Unlike covalent bonds, the behavior of these interactions varies with their energies. I systematically investigate energy-refined modeling of these interactions. Included in this is a method to assign a score to a predicted cluster decomposition, adapted from the B-cubed score from information retrieval. Another contribution of this thesis is in new approaches to measuring the robustness of rigidity analysis results. The protein\u27s fold is held in place by weak, noncovalent interactions, known to break and form during natural fluctuations. Rigidity analysis has been conventionally performed on only a single snapshot, rather than on an entire trajectory, and no information was made available on the sensitivity of the clusters to variations in the interaction network. I propose an approach to measure the robustness of rigidity results, by studying how detrimental the loss of a single interaction may be to a cluster\u27s rigidity. The accompanying study shows that, when present, highly critical interactions are concentrated around the active site, indicating that nature has designed a very versatile system for transitioning between unique conformations. Over the course of this thesis, we develop the KINARI library for experimenting with extensions to rigidity analysis. The modular design of the software allows for easy extensions and tool development. A specific feature is the inclusion of several modeling options, allowing more freedom in exploring biological hypotheses and future benchmarking experiments

Révision automatique de théories écologiques

À l’origine, ce sont des difficultés en biologie évolutive qui ont motivé cette thèse. Après des décennies à tenter de trouver une théorie basée sur la sélection capable de prédire la diversité génomique, les théoriciens n’ont pas trouvé d’alternatives pratiques à la théorie neutre. Après avoir étudié la relation entre la spéciation et la diversité (Annexes A, B, C), j’ai conclu que l’approche traditionnelle pour construire des théories serait difficile à appliquer au problème de la biodiversité. Prenons par exemple le problème de la diversité génomique, la difficulté n’est pas que l’on ignore les mécanismes impliqués, mais qu’on ne réussit pas à construire de théories capable d’intégrer ces mécanismes. Les techniques en intelligence artificielle, à l’inverse, réussissent souvent à construire des modèles prédictifs efficaces justement là où les théories traditionnelles échouent. Malheureusement, les modèles bâtis par les intelligences arti- ficielles ne sont pas clairs. Un réseau de neurones peut avoir jusqu’à un milliard de paramètres. L’objectif principal de ma thèse est d’étudier des algorithmes capable de réviser les théories écologiques. L’intégration d’idées venant de différentes branches de l’écologie est une tâche difficile. Le premier défi de ma thèse est de trouver sous quelle représentation formelle les théories écologiques doivent être encodées. Contrairement aux mathématiques, nos théories sont rarement exactes. Il y a à la fois de l’incertitude dans les données que l’on collecte, et un flou dans nos théories (on ne s’attend pas à que la théorie de niche fonctionne 100% du temps). Contrairement à la physique, où un petit nombre de forces dominent la théorie, l’écologie a un très grand nombre de théories. Le deuxième défi est de trouver comment ces théories peuvent être révisées automatiquement. Ici, le but est d’avoir la clarté des théories traditionnelles et la capacité des algorithmes en intelligence artificielle de trouver de puissants modèles prédictifs à partir de données. Les tests sont faits sur des données d’interactions d’espèces

Rigidity and Fluidity in Living and Nonliving Matter

Many of the standard equilibrium statistical mechanics techniques do not readily apply to non-equilibrium phase transitions such as the fluid-to-disordered solid transition found in repulsive particulate systems. Examples of repulsive particulate systems are sand grains and colloids. The first part of this thesis contributes to methods beyond equilibrium statistical mechanics to ultimately understand the nature of the fluid-to-disordered solid transition, or jamming, from a microscopic basis. In Chapter 2 we revisit the concept of minimal rigidity as applied to frictionless, repulsive soft sphere packings in two dimensions with the introduction of the jamming graph. Minimal rigidity is a purely combinatorial property encoded via Laman\u27s theorem in two dimensions. It constrains the global, average coordination number of the graph, for instance. Minimal rigidity, however, does not address the geometry of local mechanical stability. The jamming graph contains both properties of global mechanical stability at the onset of jamming and local mechanical stability. We demonstrate how jamming graphs can be constructed using local rules via the Henneberg construction such that these graphs are of the constraint percolation type, where percolation is the study of connected structures in disordered networks. We then probe how jamming graphs destabilize, or become fluid-like, by deleting an edge/contact in the graph and computing the resulting rigid cluster distribution. We also uncover a new potentially diverging lengthscale associated with the random deletion of contacts. In Chapter 3 we study several constraint percolation models, such as k-core percolation and counter-balance percolation, on hyperbolic lattices to better understand the role of loops in such models. The constraints in these percolation models incorporate aspects of local mechanical rigidity found in jammed systems. The expectation is that since these models are indeed easier to analyze than the more complicated problem of jamming, we will gain insight into which constraints affect the nature of the jamming transition and which do not. We find that k = 3-core percolation on the hyperbolic lattice remains a continuous phase transition despite the fact that the loop structure of hyperbolic lattices is different from Euclidean lattices. We also contribute towards numerical techniques for analyzing percolation on hyperbolic lattices. In Chapters 4 and 5 we turn to living matter, which is also nonequilibrium in a very local way in that each constituent has its own internal energy supply. In Chapter 4 we study the fluidity of a cell moving through a confluent tissue, i.e. a group of cells with no gaps between them, via T1 transitions. A T1 transition allows for an edge swap so that a cell can come into contact with new neighbors. Cell migration is then generated by a sequence of such swaps. In a simple four cell system we compute the energy barriers associated with this transition. We then find that the energy barriers in a larger system are rather similar to the four cell case. The many cell case, however, more easily allows for the collection of statistics of these energy barriers given the disordered packings of cell observed in experiments. We find that the energy barriers are exponentially distributed. Such a finding implies that glassy dynamics is possible in a confluent tissue. Finally, in chapter 5 we turn to single cell migration in the extracellular matrix, another native environment of a cell. Experiments suggest that the migration of some cells in the three-dimensional ext ra cellular matrix bears strong resemblance to one-dimensional cell migration. Motivated by this observation, we construct and study a minimal one-dimensional model cell made of two beads and an active spring moving along a rigid track. The active spring models the stress fibers with their myosin-driven contractility and alpha-actinin-driven extendability, while the friction coefficients of the two beads describe the catch/slip bond behavior of the integrins in focal adhesions. Net motion arises from an interplay between active contractility (and passive extendability) of the stress fibers and an asymmetry between the front and back of the cell due to catch bond behavior of integrins at the front of the cell and slip bond behavior of integrins at the back. We obtain reasonable cell speeds with independently estimated parameters. Our model highlights the role of alpha-actinin in three-dimensional cell motility and does not require Arp2/3 actin filament nucleation for net motion