Exploring the grammar of perception A case study using data from Russian

In this paper, I pursue the distributional hypothesis that the meaning of a word is derived from the linguistic contexts in which it occurs and apply it to verbs of perception. Differently from NLP implementations of the distributional hypothesis, I explicitly limit the range of variables to the grammatical domain and chart the way in which verbs of Vision, Hearing and Touch are used, morphologically and syntactically, in a representative sample of corpus data. Some aspects of experience are so central and pervasive that reference to them has grammaticalized (Divjak 2010; see also Janda & Lyashevskaya 2011; Newman 2008). The aim is, firstly, to determine to which extent a verb’s grammatical context alone allows us to classify utterances according to the perception type, and, secondly, to chart the similarities and differences in the verbs’ preference for morphological markers and syntactic constructions. If contexts are highly specialized, language structure, as it is witnessed in use, could assist sensory impaired speakers in building up viable representations of concepts, even if sensory experience is lacking. If, in addition, similarities between certain sensory perception verbs are high, sensory impaired speakers could use these similarities to perform analogical mapping across senses and ground concepts relating to the impaired sense in a cognate sensory experience. The findings are relevant for concept acquisition and representation in general and for concept acquisition and representation in sensory impaired populations, such as the blind, in particular

Alignment of speech and co-speech gesture in a constraint-based grammar

This thesis concerns the form-meaning mapping of multimodal communicative actions consisting of speech signals and improvised co-speech gestures, produced spontaneously with the hand. The interaction between speech and speech-accompanying gestures has been standardly addressed from a cognitive perspective to establish the underlying cognitive mechanisms for the synchronous speech and gesture production, and also from a computational perspective to build computer systems that communicate through multiple modalities. Based on the findings of this previous research, we advance a new theory in which the mapping from the form of the combined speech-and-gesture signal to its meaning is analysed in a constraint-based multimodal grammar. We propose several construction rules about multimodal well-formedness that we motivate empirically from an extensive and detailed corpus study. In particular, the construction rules use the prosody, syntax and semantics of speech, the form and meaning of the gesture signal, as well as the temporal performance of the speech relative to the temporal performance of the gesture to constrain the derivation of a single multimodal syntax tree which in turn determines a meaning representation via standard mechanisms for semantic composition. Gestural form often underspecifies its meaning, and so the output of our grammar is underspecified logical formulae that support the range of possible interpretations of the multimodal act in its final context-of-use, given the current models of the semantics/ pragmatics interface. It is standardly held in the gesture community that the co-expressivity of speech and gesture is determined on the basis of their temporal co-occurrence: that is, a gesture signal is semantically related to the speech signal that happened at the same time as the gesture. Whereas this is usually taken for granted, we propose a methodology of establishing in a systematic and domain-independent way which spoken element(s) gesture can be semantically related to, based on their form, so as to yield a meaning representation that supports the intended interpretation(s) in context. The ‘semantic’ alignment of speech and gesture is thus driven not from the temporal co-occurrence alone, but also from the linguistic properties of the speech signal gesture overlaps with. In so doing, we contribute a fine-grained system for articulating the form-meaning mapping of multimodal actions that uses standard methods from linguistics. We show that just as language exhibits ambiguity in both form and meaning, so do multimodal actions: for instance, the integration of gesture is not restricted to a unique speech phrase but rather speech and gesture can be aligned in multiple multimodal syntax trees thus yielding distinct meaning representations. These multiple mappings stem from the fact that the meaning as derived from gesture form is highly incomplete even in context. An overall challenge is thus to account for the range of possible interpretations of the multimodal action in context using standard methods from linguistics for syntactic derivation and semantic composition

Algebraic Stream Processing

We identify and analyse the typically higher-order approaches to stream processing in the literature. From this analysis we motivate an alternative approach to the specification of SPSs as STs based on an essentially first-order equational representation. This technique is called Cartesian form specification. More specifically, while STs are properly second-order objects we show that using Cartesian forms, the second-order models needed to formalise STs are so weak that we may use and develop well-understood first-order methods from computability theory and mathematical logic to reason about their properties. Indeed, we show that by specifying STs equationally in Cartesian form as primitive recursive functions we have the basis of a new, general purpose and mathematically sound theory of stream processing that emphasises the formal specification and formal verification of STs. The main topics that we address in the development of this theory are as follows. We present a theoretically well-founded general purpose stream processing language ASTRAL (Algebraic Stream TRAnsformer Language) that supports the use of modular specification techniques for full second-order STs. We show how ASTRAL specifications can be given a Cartesian form semantics using the language PREQ that is an equational characterisation of the primitive recursive functions. In more detail, we show that by compiling ASTRAL specifications into an equivalent Cartesian form in PREQ we can use first-order equational logic with induction as a logical calculus to reason about STs. In particular, using this calculus we identify a syntactic class of correctness statements for which the verification of ASTRAL programmes is decidable relative to this calculus. We define an effective algorithm based on term re-writing techniques to implement this calculus and hence to automatically verify a very broad class of STs including conventional hardware devices. Finally, we analyse the properties of this abstract algorithm as a proof assistant and discuss various techniques that have been adopted to develop software tools based on this algorithm

Systems between information and knowledge : In a memory management model of an extended enterprise

The research question of this thesis was how knowledge can be managed with information systems. Information systems can support but not replace knowledge management. Systems can mainly store epistemic organisational knowledge included in content, and process data and information. Certain value can be achieved by adding communication technology to systems. All communication, however, can not be managed. A new layer between communication and manageable information was named as knowformation. Knowledge management literature was surveyed, together with information species from philosophy, physics, communication theory, and information system science. Positivism, post-positivism, and critical theory were studied, but knowformation in extended organisational memory seemed to be socially constructed. A memory management model of an extended enterprise (M3.exe) and knowformation concept were findings from iterative case studies, covering data, information and knowledge management systems. The cases varied from groups towards extended organisation. Systems were investigated, and administrators, users (knowledge workers) and managers interviewed. The model building required alternative sets of data, information and knowledge, instead of using the traditional pyramid. Also the explicit-tacit dichotomy was reconsidered. As human knowledge is the final aim of all data and information in the systems, the distinction between management of information vs. management of people was harmonised. Information systems were classified as the core of organisational memory. The content of the systems is in practice between communication and presentation. Firstly, the epistemic criterion of knowledge is not required neither in the knowledge management literature, nor from the content of the systems. Secondly, systems deal mostly with containers, and the knowledge management literature with applied knowledge. Also the construction of reality based on the system content and communication supports the knowformation concept. Knowformation belongs to memory management model of an extended enterprise (M3.exe) that is divided into horizontal and vertical key dimensions. Vertically, processes deal with content that can be managed, whereas communication can be supported, mainly by infrastructure. Horizontally, the right hand side of the model contains systems, and the left hand side content, which should be independent from each other. A strategy based on the model was defined.Tutkimuksen tavoitteena oli määrittää, miten tietojärjestelmiä voidaan käyttää organisaatioiden tietämyksen hallintaan. Johtopäätöksenä voidaan sanoa, että järjestelmillä voidaan tukea, mutta ei korvata tietojohtamista. Tietojärjestelmiä voidaan käyttää lähinnä organisaation episteemisen tiedon muistina, prosessoitavan tiedon varastointiin. Oleellista lisäarvoa saadaan, jos viestintäteknologiaa käytetään tietojärjestelmien tukena. Kommunikaatiota ei kuitenkaan voida johtaa, sillä se ei perustu prosesseihin, vaan enintään työnkulkuun ja sitä vapaampaan viestintään. Hallitun informaation ja viestinnän välille syntyy knowformaatioksi nimetty kerros, lähinnä organisaatioiden lyhytkestoiseen muistiin. Uusi knowformaatio-käsite on käytännön tapaustutkimusten tulos. Vastaavaa ei aiemmissa tietojohtamisen tutkimuksissa ole esitetty. Tietojohtamisen kirjallisuuden taustaksi tutkittiin fysiikan, filosofian, viestinnän ja tietojenkäsittelytieteen luokitukset. Tapaustutkimuksissa tarkasteltiin useita datan hallinnan, dokumentaation ja tietojohtamisen järjestelmiä organisaation sisäisissä ryhmissä, organisaation laajuisesti sekä organisaation yhteistyökumppaneiden kanssa. Tapauksissa tutkittiin niin järjestelmien ominaisuudet kuin myös eri sidosryhmien kokemukset. Tutkimuksessa tietojärjestelmät luokiteltiin organisaation muistin ytimeen. Knowformaation kerrosta tarvitaan toisaalta koska filosofisen tiedon episteemistä kriteeriä ei edellytetä järjestelmien sisällöltä (eikä tietojohtamisen kirjallisuuden käsitemäärittelyissä) ja toisaalta koska tiedon uudelleenkonstruoinnissa merkitys muuttuu. Tulevien järjestelmien suunnitteluun tarvitaan uusi näkökulma, koska data, informaatio ja knowledge tasojen hierarkia ei erotu eri järjestelmätyyppien käyttäjien sosiaalisesti konstruoidussa todellisuudessa. Tieteen filosofian skaala positivistisesta konstruktivistiseen oli mallin muodostuksessa oleellinen, ja sen validiuden todentamisen jälkeen eksplisiittinen piiloinen -dikotomia mallinettiin uudelleen knowformaatio-käsitteen avulla. Uusi tietomalli ja knowformaatio-käsite tarvitaan työn päätuloksessa, jatketun organisaation muistin hallintamallissa. Sen ääripäihin kuluvat kommunikaatio, jota tuetaan, ja toisessa päässä prosessit, joita hallitaan. Kahden muun entiteetin, järjestelmien ja niiden sisällön, tulisi olla riippumattomia toisistaan. Knowformaatio elää näiden kokonaisuuksien implisiittisillä rajoilla, informaation ja tiedon välisellä harmaalla alueella

Hierarchical Reinforcement Learning for Spoken Dialogue Systems

Institute for Communicating and Collaborative SystemsThis thesis focuses on the problem of scalable optimization of dialogue behaviour in speech-based conversational systems using reinforcement learning. Most previous investigations in dialogue strategy learning have proposed flat reinforcement learning methods, which are more suitable for small-scale spoken dialogue systems. This research formulates the problem in terms of Semi-Markov Decision Processes (SMDPs), and proposes two hierarchical reinforcement learning methods to optimize sub-dialogues rather than full dialogues. The first method uses a hierarchy of SMDPs, where every SMDP ignores irrelevant state variables and actions in order to optimize a sub-dialogue. The second method extends the first one by constraining every SMDP in the hierarchy with prior expert knowledge. The latter method proposes a learning algorithm called 'HAM+HSMQ-Learning', which combines two existing algorithms in the literature of hierarchical reinforcement learning. Whilst the first method generates fully-learnt behaviour, the second one generates semi-learnt behaviour. In addition, this research proposes a heuristic dialogue simulation environment for automatic dialogue strategy learning. Experiments were performed on simulated and real environments based on a travel planning spoken dialogue system. Experimental results provided evidence to support the following claims: First, both methods scale well at the cost of near-optimal solutions, resulting in slightly longer dialogues than the optimal solutions. Second, dialogue strategies learnt with coherent user behaviour and conservative recognition error rates can outperform a reasonable hand-coded strategy. Third, semi-learnt dialogue behaviours are a better alternative (because of their higher overall performance) than hand-coded or fully-learnt dialogue behaviours. Last, hierarchical reinforcement learning dialogue agents are feasible and promising for the (semi) automatic design of adaptive behaviours in larger-scale spoken dialogue systems. This research makes the following contributions to spoken dialogue systems which learn their dialogue behaviour. First, the Semi-Markov Decision Process (SMDP) model was proposed to learn spoken dialogue strategies in a scalable way. Second, the concept of 'partially specified dialogue strategies' was proposed for integrating simultaneously hand-coded and learnt spoken dialogue behaviours into a single learning framework. Third, an evaluation with real users of hierarchical reinforcement learning dialogue agents was essential to validate their effectiveness in a realistic environment

The feasibility of electronic journals: some studies in human–computer interaction

Computer-based tools for communication are a recent technological development. They promise to provide new routes by which to communicate with others and to transform some communications that have hitherto been dependent on media such as paper. One example is the possibility of supporting scholarly communication by the use of electronic systems, which also promises a method by which the information explosion might be handled. The research is an examinat4on of whether or not the support of scholarly communication in this way is feasible. To investigate communication systems requires a large scale study over a long period. Accordingly the research rests on a study programme on 'electronic journals', BLEND, which ran from 1980 to 1984, funded by the British Library Research and Development Department. The feasibility of ielectronic journals is investigated by exploring the usability, utility, likeability and cost-effectiveness of the communications system. An analysis of the frequency and distribution of the use of the computer-based communications system showed that many things seemed to get in the way of accessing it. Several techniques were used to examine this: transaction recording, interviews, telephone surveys, questionnaires and analysis of requests for help. Once the system was accessed, a comparison of users' aims with actual use shows that different forms of the journal should be explored in the future. Two reasons for the access rate and type of use made of the system was the degree to which researchers were able to accommodate the use of a new communications system into existing patterns of work and the level of usability of the system. One area in usability that is explored in detail is the way that text can be read easily on a screen. The cost-effectiveness of the system is examined by projecting from actual costs and patterns of use. The final chapters bring together the studies in a 'Barrier' framework for understanding the use of a communications system and look forward to the future of electronic journals

Third International Symposium on Space Mission Operations and Ground Data Systems, part 1

Under the theme of 'Opportunities in Ground Data Systems for High Efficiency Operations of Space Missions,' the SpaceOps '94 symposium included presentations of more than 150 technical papers spanning five topic areas: Mission Management, Operations, Data Management, System Development, and Systems Engineering. The papers focus on improvements in the efficiency, effectiveness, productivity, and quality of data acquisition, ground systems, and mission operations. New technology, techniques, methods, and human systems are discussed. Accomplishments are also reported in the application of information systems to improve data retrieval, reporting, and archiving; the management of human factors; the use of telescience and teleoperations; and the design and implementation of logistics support for mission operations