Learning in Markov Random Fields with Contrastive Free Energies

Learning Markov random field (MRF) models is notoriously hard due to the presence of a global normalization factor. In this paper we present a new framework for learning MRF models based on the contrastive free energy (CF) objective function. In this scheme the parameters are updated in an attempt to match the average statistics of the data distribution and a distribution which is (partially or approximately) "relaxed" to the equilibrium distribution. We show that maximum likelihood, mean field, contrastive divergence and pseudo-likelihood objectives can be understood in this paradigm. Moreover, we propose and study a new learning algorithm: the "kstep Kikuchi/Bethe approximation". This algorithm is then tested on a conditional random field model with "skip-chain" edges to model long range interactions in text data. It is demonstrated that with no loss in accuracy, the training time is brought down on average from 19 hours (BP based learning) to 83 minutes, an order of magnitude improvement

Wormhole Hamiltonian Monte Carlo

In machine learning and statistics, probabilistic inference involving multimodal distributions is quite difficult. This is especially true in high dimensional problems, where most existing algorithms cannot easily move from one mode to another. To address this issue, we propose a novel Bayesian inference approach based on Markov Chain Monte Carlo. Our method can effectively sample from multimodal distributions, especially when the dimension is high and the modes are isolated. To this end, it exploits and modifies the Riemannian geometric properties of the target distribution to create \emph{wormholes} connecting modes in order to facilitate moving between them. Further, our proposed method uses the regeneration technique in order to adapt the algorithm by identifying new modes and updating the network of wormholes without affecting the stationary distribution. To find new modes, as opposed to rediscovering those previously identified, we employ a novel mode searching algorithm that explores a \emph{residual energy} function obtained by subtracting an approximate Gaussian mixture density (based on previously discovered modes) from the target density function

Learning generative models of mid-level structure in natural images

Natural images arise from complicated processes involving many factors of variation. They reflect the wealth of shapes and appearances of objects in our three-dimensional world, but they are also affected by factors such as distortions due to perspective, occlusions, and illumination, giving rise to structure with regularities at many different levels. Prior knowledge about these regularities and suitable representations that allow efficient reasoning about the properties of a visual scene are important for many image processing and computer vision tasks. This thesis focuses on models of image structure at intermediate levels of complexity as required, for instance, for image inpainting or segmentation. It aims at developing generative, probabilistic models of this kind of structure, and, in particular, at devising strategies for learning such models in a largely unsupervised manner from data. One hallmark of natural images is that they can often be decomposed into regions with very different visual characteristics. The main approach of this thesis is therefore to represent images in terms of regions that are characterized by their shapes and appearances, and an image is then composed from many such regions. We explore approaches to learn about the appearance of regions, to learn about region shapes, and ways to combine several regions to form a full image. To achieve this goal, we make use of some ideas for unsupervised learning developed in the literature on models of low-level image structure and in the “deep learning” literature. These models are used as building blocks of more structured model formulations that incorporate additional prior knowledge of how images are formed. The thesis makes the following contributions: Firstly, we investigate a popular, MRF based prior of natural image structure, the Field-of Experts, with respect to its ability to model image textures, and propose an extended formulation that is considerably more successful at this task. This formulation gives rise to a fully parametric, translation-invariant probabilistic generative model of image textures. We illustrate how this model can be used as a component of a more comprehensive model of images comprising multiple textured regions. Secondly, we develop a model of region shape. This work is an extension of the “Masked Restricted Boltzmann Machine” proposed by Le Roux et al. (2011) and it allows explicit reasoning about the independent shapes and relative depths of occluding objects. We develop an inference and unsupervised learning scheme and demonstrate how this shape model, in combination with the masked RBM gives rise to a good model of natural image patches. Finally, we demonstrate how this model of region shape can be extended to model shapes in large images. The result is a generative model of large images which are formed by composition from many small, partially overlapping and occluding objects

Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense

Recent progress in deep learning is essentially based on a "big data for small tasks" paradigm, under which massive amounts of data are used to train a classifier for a single narrow task. In this paper, we call for a shift that flips this paradigm upside down. Specifically, we propose a "small data for big tasks" paradigm, wherein a single artificial intelligence (AI) system is challenged to develop "common sense", enabling it to solve a wide range of tasks with little training data. We illustrate the potential power of this new paradigm by reviewing models of common sense that synthesize recent breakthroughs in both machine and human vision. We identify functionality, physics, intent, causality, and utility (FPICU) as the five core domains of cognitive AI with humanlike common sense. When taken as a unified concept, FPICU is concerned with the questions of "why" and "how", beyond the dominant "what" and "where" framework for understanding vision. They are invisible in terms of pixels but nevertheless drive the creation, maintenance, and development of visual scenes. We therefore coin them the "dark matter" of vision. Just as our universe cannot be understood by merely studying observable matter, we argue that vision cannot be understood without studying FPICU. We demonstrate the power of this perspective to develop cognitive AI systems with humanlike common sense by showing how to observe and apply FPICU with little training data to solve a wide range of challenging tasks, including tool use, planning, utility inference, and social learning. In summary, we argue that the next generation of AI must embrace "dark" humanlike common sense for solving novel tasks.Comment: For high quality figures, please refer to http://wellyzhang.github.io/attach/dark.pd

‘Dobraia Staraia Angliia’ in Russian Perception: Literary Representations of Englishness in Translated Children's Literature in Soviet and Post-Soviet Russia

This thesis explores Englishness and its representation in translated children’s literature in Russia during the Soviet period (from 1917 until 1991) and the post-Soviet period (from 1992 until 2015). It focuses on Russian translations of English children’s classics published between the late-Victorian period and the Second World War. It studies how Russian translations of English children’s literature construct literary portrayals of Englishness in varied socio-cultural and historical contexts. It investigates the complex processes involved in re-creating national specificities of English literary texts in Russian culture. The Anglo-centric essence of Englishness – or ‘dobraia staraia Angliia’ [good old England] – is expressed to a greater degree in the classics of English children’s literature. It is this particular idealised Englishness that is represented in the Russian translations. This thesis demonstrates that various manifestations of Englishness are modified in Russian translations and that the degree of modification varies according to changes in the political climate in Russia. A significant role is played by ideology – of a prevailing political nature during in the Soviet Union and a commercial ideology in post-Soviet Russia. The first chapter lays the theoretical foundation for the whole thesis and outlines the methodology adopted. Chapters 2 and 3 set out the contextual background for understanding Englishness by focusing on the question of Englishness perceived from English and Russian perspectives, and discussing the main tendencies of representing Englishness in both cultures. Chapter 4 presents the historical background by highlighting the political and cultural circumstances in which Russian translations were made. The second half of the thesis (chapters five, six and seven) focuses on the analysis of the representation of Englishness in Russian translations. Chapter 5 discusses which English children’s books, published between the late-Victorian period and the Second World War, were selected for translation and at what point between 1918 and 2015. Chapters 6 and 7 present the case studies in this thesis. These provide an analysis of how different manifestations of Englishness were translated and, taking into account the Soviet and post-Soviet historical contexts, examine why they were translated in certain ways.Arts and Humanities Research Counci

Creating Digital Editions for Corpus Linguistics : The case of Potage Dyvers, a family of six Middle English recipe collections

This thesis presents a corpus-linguistically oriented digital documentary edition of six 15th-century culinary recipe collections, known as the Potage Dyvers family, with an introduction to its historical context and an analysis of its dialectal and structural features, and defines an editorial framework for producing such editions for the purposes of corpus linguistic research. Traditionally historical corpora have been compiled from printed editions not originally designed to serve as corpus linguistic data. Recently, both the digitalisation of textual editing and the turning of corpus compilers towards original sources have blurred the boundaries between these two crafts, placing corpus compilers into an editorial role. Despite the fact that traditional editorial approaches have been recognised as largely incompatible with the needs of linguistic research, and the established methods of corpus encoding do not satisfactorily represent the documentary context of manuscript texts, no explicitly linguistic editorial approach has so far been designed for editing manuscript sources for use in corpora. Even most digital editions, despite their advanced representational capabilities, are literary or historical in orientation and thus do not provide an adequate model. The editorial framework described here and the edition based on it have been explicitly designed to answer the needs of historical corpus linguistics. First, it aims at faithfully modelling the manuscript as a historical artefact, including both its textual content and its visual and material paratext, whose communicative importance has also been recognised by many historical linguists. Second, it presents this model in a form which allows not only the study of both text and paratext using corpus linguistic methods, but also allows resulting analytical metadata to be linked back to the edition, shared with other scholars, and used as the basis for further study. The edition itself is provided as a digital appendix to the thesis in the form of both a digital data archive encoded in TEI XML and three editorial presentations of this data, and serves not only as a demonstration of the editorial approach, but also provides a valuable new research resource. The choice of material is based on the insight that utilitarian texts like recipes provide valuable material especially for historical pragmatics and discourse studies. As one of the first vernacular text types, recipes also provide an excellent opportunity to study the diachronic development of a single textual genre. The Potage Dyvers family is the second largest known family of Middle English recipe collections, surviving in six physically diverse manuscripts. Of these, four were edited in 1888 by conflating them into two collections, but their complex interrelationships have so far escaped systematic study. The structural analysis of the six Potage Dyvers versions indicates that the family, containing a total of 371 unique recipes, in fact consists of three sibling pairs of MSS. Two of these contain largely the same material but in a different order, while the third shares only a core of 89 recipes with the others, deriving a large number of recipes from other sources. In terms of their language, all of the six versions exhibit mainly Midlands forms and combine dialectally unmarked forms with more local variants from different areas, reflecting the 15th-century loss of dialectal distinctions which has not yet reached orthographic or morphological uniformity, and indicating possible metropolitan associations.Tämä väitöskirja tarjoaa korpuslingvistisesti suuntautuneen digitaalisen tekstiedition kuudesta samankaltaisesta 1400-luvun englanninkielisestä ruokareseptikokoelmasta, jotka tunnetaan nimellä Potage Dyvers. Väitöskirja sisältää johdannon tekstien historialliseen kontekstiin sekä murrepiirteisiin ja tekstirakenteeseen pohjautuvat analyysit niiden todennäköisestä alkuperästä ja keskinäisistä suhteista. Väitöskirja kartoittaa historiallisen kielentutkimuksen käsikirjoituseditiolle asettamat vaatimukset ja määrittelee yksityiskohtaisen ohjeiston niiden täyttämiseksi. Historialliset tekstikorpukset on perinteisesti koottu digitoimalla painettuja tekstieditioita joita ei ole suunniteltu kielitieteelliseksi aineistoksi. Viime vuosina tekstieditioiden digitaalistuminen ja korpuslingvistien lisääntynyt kiinnostus alkuperäisiä dokumenttilähteitä kohtaan ovat häivyttäneet tekstieditoinnin ja kielikorpusten kokoamisen välistä rajaa. Vaikka yhtäältä perinteisten editointimenetelmien ongelmat kielentutkimuksen suhteen ja toisaalta aiempien historiallisten kielikorpusten tapa jättää huomiotta käsikirjoitustekstien materiaalinen konteksti on havaittu ongelmallisiksi, ei historiallisten käsikirjoituslähteiden esittämiseen tekstikorpuksissa ole kehitetty juurikaan menetelmiä. Väitöskirjan sisältämä ja kuvaama editio on suunniteltu erityisesti historiallisen korpuslingvistiikan tarpeisiin. Se pyrkii mallintamaan käsikirjoituksen historiallisena esineenä, tallentaen digitaalisesti paitsi tekstin, myös sen viestinnällisen merkityksen kannalta olennaisen materiaalisen kontekstin. Tämä malli esitetään muodossa, joka mahdollistaa paitsi sekä tekstin että materiaalisen kontekstin tutkimisen korpusmenetelmin, myös tutkimuksen tuloksena syntyvän metatiedon liittämisen alkuperäiseen editioon ja käyttämisen myöhemmän tutkimuksen pohjana. Itse editio joka toimii paitsi esimerkkinä editointimenetelmän käytöstä, myös itsessään arvokkaana tutkimusaineistona sisältyy väitöskirjan digitaalisiin liitteisiin sekä TEI XML -muotoisena digitaalisena data-arkistona että kolmessa erilaisessa esitysmuodossa. Keskiaikaiset reseptitekstit on valittu edition aineistoksi, koska niiden kaltaiset käytännölliset tekstit ovat arvokasta materiaalia esimerkiksi historialliselle pragmatiikalle ja diskurssintutkimukselle. Yhtenä vanhimmista kansankielisistä tekstilajeista reseptit myös tarjoavat erinomaisen tilaisuuden yksittäisen tekstilajin historiallisen kehityksen tutkimiseen. Kuutena eri versiona säilynyt Potage Dyvers on toiseksi suurin keskienglanninkielisten reseptikokoelmien ryhmä. Sen neljästä versiosta on olemassa vuonna 1888 julkaistu editio jossa ne esitettiin kahtena erillisenä tekstinä, mutta versioiden välisiä monimutkaisia suhteita ei ole tutkittu järjestelmällisesti. Versioiden välinen rakenneanalyysi osoittaa, että tämä yhteensä 371 ainutkertaista reseptiä sisältävä ryhmä koostuu itse asiassa kolmesta keskenään samankaltaisten kokoelmien parista. Näistä pareista kaksi sisältävät pääosin samat reseptit mutta hyvin eri järjestyksessä, kun taas kolmas jakaa muiden kanssa vain 89 reseptiä joihin se yhdistää suuren määrän reseptejä muista lähteistä. Kieleltään kaikki kuusi versiota edustavat pääosin Midlandsin alueen kielimuotoa, mutta murteellisesti värittömien muotojen suosiminen ja yhdistyminen useiden eri alueiden paikallisiin piirteisiin heijastaa 1400-luvulla tapahtunutta kieliasun yhtenäistymistä edeltävää murre-erojen tasaantumista, ja on erityisen tyypillistä Lontoon suurkaupunkialueen kielimuodolle