Detection and Evaluation of Clusters within Sequential Data

Motivated by theoretical advancements in dimensionality reduction techniques we use a recent model, called Block Markov Chains, to conduct a practical study of clustering in real-world sequential data. Clustering algorithms for Block Markov Chains possess theoretical optimality guarantees and can be deployed in sparse data regimes. Despite these favorable theoretical properties, a thorough evaluation of these algorithms in realistic settings has been lacking. We address this issue and investigate the suitability of these clustering algorithms in exploratory data analysis of real-world sequential data. In particular, our sequential data is derived from human DNA, written text, animal movement data and financial markets. In order to evaluate the determined clusters, and the associated Block Markov Chain model, we further develop a set of evaluation tools. These tools include benchmarking, spectral noise analysis and statistical model selection tools. An efficient implementation of the clustering algorithm and the new evaluation tools is made available together with this paper. Practical challenges associated to real-world data are encountered and discussed. It is ultimately found that the Block Markov Chain model assumption, together with the tools developed here, can indeed produce meaningful insights in exploratory data analyses despite the complexity and sparsity of real-world data.Comment: 37 pages, 12 figure

Content-based video indexing for sports applications using integrated multi-modal approach

This thesis presents a research work based on an integrated multi-modal approach for sports video indexing and retrieval. By combining specific features extractable from multiple (audio-visual) modalities, generic structure and specific events can be detected and classified. During browsing and retrieval, users will benefit from the integration of high-level semantic and some descriptive mid-level features such as whistle and close-up view of player(s). The main objective is to contribute to the three major components of sports video indexing systems. The first component is a set of powerful techniques to extract audio-visual features and semantic contents automatically. The main purposes are to reduce manual annotations and to summarize the lengthy contents into a compact, meaningful and more enjoyable presentation. The second component is an expressive and flexible indexing technique that supports gradual index construction. Indexing scheme is essential to determine the methods by which users can access a video database. The third and last component is a query language that can generate dynamic video summaries for smart browsing and support user-oriented retrievals

Holistic Vocabulary Independent Spoken Term Detection

Within this thesis, we aim at designing a loosely coupled holistic system for Spoken Term Detection (STD) on heterogeneous German broadcast data in selected application scenarios. Starting from STD on the 1-best output of a word-based speech recognizer, we study the performance of several subword units for vocabulary independent STD on a linguistically and acoustically challenging German corpus. We explore the typical error sources in subword STD, and find that they differ from the error sources in word-based speech search. We select, extend and combine a set of state-of-the-art methods for error compensation in STD in order to explicitly merge the corresponding STD error spaces through anchor-based approximate lattice retrieval. Novel methods for STD result verification are proposed in order to increase retrieval precision by exploiting external knowledge at search time. Error-compensating methods for STD typically suffer from high response times on large scale databases, and we propose scalable approaches suitable for large corpora. Highest STD accuracy is obtained by combining anchor-based approximate retrieval from both syllable lattice ASR and syllabified word ASR into a hybrid STD system, and pruning the result list using external knowledge with hybrid contextual and anti-query verification.Die vorliegende Arbeit beschreibt ein lose gekoppeltes, ganzheitliches System zur Sprachsuche auf heterogenenen deutschen Sprachdaten in unterschiedlichen Anwendungsszenarien. Ausgehend von einer wortbasierten Sprachsuche auf dem Transkript eines aktuellen Wort-Erkenners werden zunächst unterschiedliche Subwort-Einheiten für die vokabularunabhängige Sprachsuche auf deutschen Daten untersucht. Auf dieser Basis werden die typischen Fehlerquellen in der Subwort-basierten Sprachsuche analysiert. Diese Fehlerquellen unterscheiden sich vom Fall der klassichen Suche im Worttranskript und müssen explizit adressiert werden. Die explizite Kompensation der unterschiedlichen Fehlerquellen erfolgt durch einen neuartigen hybriden Ansatz zur effizienten Ankerbasierten unscharfen Wortgraph-Suche. Darüber hinaus werden neuartige Methoden zur Verifikation von Suchergebnissen vorgestellt, die zur Suchzeit verfügbares externes Wissen einbeziehen. Alle vorgestellten Verfahren werden auf einem umfangreichen Satz von deutschen Fernsehdaten mit Fokus auf ausgewählte, repräsentative Einsatzszenarien evaluiert. Da Methoden zur Fehlerkompensation in der Sprachsuchforschung typischerweise zu hohen Laufzeiten bei der Suche in großen Archiven führen, werden insbesondere auch Szenarien mit sehr großen Datenmengen betrachtet. Die höchste Suchleistung für Archive mittlerer Größe wird durch eine unscharfe und Anker-basierte Suche auf einem hybriden Index aus Silben-Wortgraphen und silbifizierter Wort-Erkennung erreicht, bei der die Suchergebnisse mit hybrider Verifikation bereinigt werden

Computational analysis of Drosophila courtship behaviour

Die Taufliege Drosophila melanogaster ist ein weitverbreiteter Modellorganismus für Studien in Molekularbiologie und Gehirnforschung. Man assoziiert sie mit einer grossen Ansammlung von Werkzeugen fuer genetische Manipulationen und mit stabilem angeborenem Verhalten, insbesonders das Balzverhalten ist ein robustes und geschlechtsspezifisches Verhalten welches anatomische und funktionale Analysen von neuronalen Schaltkreisen ermöglicht. Die Arbeit in dieser Dissertation stellt ein eine automatische Quantifizierung für funktionale Verhaltensanalysen vor die es ermöglicht Verhaltensunterschiede zwischen Fliegen mit genetisch manipulierten Neuronen oder neuronalen Schaltkreisen zu messen und zu visualisieren. Da Genetiker typischerweise eine grosse Anzahl von Experimenten durchführen wurde die Automatisierung des Analyseprozesses von Serien von Verhaltensexperimenten angestrebt. Ein automatisches Analysewerkzeug bringt viele Vorteile, es spart Zeit, reduziert menschliche Fehler und bringt eine Erweiterung der Analysemöglichkeiten. Eine automatische Quantifizierung von beobarchtetem Verhalten bringt weiters robuste und reproduzierbare Analysen. Die Quantifizierung selbst erfolg in zwei Schritten, einem Bildverarbeitungsschritt in welchem aufgezeichnete Verhaltensvideos in eine Zeitreihe uebersetzt werden und einem Mustererkennungsschritt in welchem eine solche Zeitreiche nach bekannten oder gelernten Mustern durchsucht wird. Das vorgestellte system bietet Lösungen fuer alle Schluesselherausforderungen einer solchen automatischen Quantifizierung, im speziellen für eine automatische Arenadetektierung, einen automatische Qualitätskontrolle für Videos , die Segmentierung der Fliegen, das Auflösen bzw. Zuordnen von Überdeckungen, das Identifizieren des Kopfendes und das Detektieren von biologisch relevanten Ereignissen. Speziell das Auflösen von Überdeckungen hat sich als eine wichtige, aber schwierige Aufgabe herausgestellt, es wurde viel Energie investiert um auch diese Herausforderung in Angriff zu nehmen und zu lösen. Das Ergebnis ist ein vollautomatisches System welches einen Rahmen fuer supervised learning beinhaltet, welcher das Trainieren und Anwenden von bottom-up Klassifikatoren, z. B. für Balzverhalten, ermöglicht. Das System beinhaltet weiters geschlechtsspezifische und fru-abhängige top-down Klassifikatoren welche die automatische Identifikation von einzelnen Balz-Schritten ermöglichen. Das System wurde so entworfen dass Benutzerinteraktionen minimiert werden, es arbeitet alle involvierten Bildverarbeitungs- und Analyseschritte vollautomatisch ab und erlaubt daher eine robuste und objektive Analyse mit hohem Durchsatz, und ist somit anwendbar für die Analyse grosser Mengen von Videodaten. Das modulare Design des systems erlaubt weiters das Entwickeln von Spin-offs, welche den Bildverarbeitungsteil wiederverwenden und darauf aufbauend Verhaltensanalysen berechnen, die speziell auf andere Biologische Experimentumgebungen oder neue Verhaltensmuster abgestimmt sind. Im Hauptanwendungsfall, in dem das Balzverhalten der Drosophila melanogaster analysiert wird, werden automatisch Ethogramme generiert, die das beobachtete Balzverhalten quantifizieren und visualisieren.The fruit fly Drosophila melanogaster is a well established model organism for molecular biology and neuroscience studies. It comes with a large set of genetic tools and well preserved innate behaviours, in particular courtship behaviour is a robust and sexually dimorphic behaviour which enables anatomical and functional studies of neuronal circuits. This thesis is about an automated quantification for functional behaviour studies, it enables measurement and visualization of differences in behaviour for flies with genetically manipulated neurons and neuronal circuits. Since geneticists typically do large numbers of experiments the aim was to automate the analysis process for a behaviour screen. There are multiple benefits gained from an automated tool like saving time, limiting human error and extending possibilities of analysis. Automated quantification of observed behaviour also provides robust and therefore reproducible analysis. The quantification itself comes in two steps, an image processing step where recorded fly videos are translated into a time series and a pattern recognition step which searches for known or learned patterns within that time series. The proposed system offers solutions for all key challenges that were encountered for automated quantification, in particular for arena detection, video quality control, fly segmentation, occlusion resolvement, heading resolvement and event detection. Especially resolving occlusions turned out to be an important but difficult task, therefore a lot of energy was invested to attack that particular challenge. The result is a fully automated system containing a supervised learning framework that allows to train and apply bottom-up classifiers, e.g. for courtship behaviour, and sex-specific and fru-dependent top-down classifiers that automatically identify individual courtship steps. The system is designed to minimize user interaction and therefore performs all involved video processing and analysis steps in a fully automated way, it therefore enables a robust, objective and high throughput analysis of large amounts of video data. The modular structure of the system allows generating spin-o¤ trackers that reuse the system's image processing part for different downstream analysis parts that may be specifically designed or adapted for new biological assays. For the standard courtship assay the system automatically computes ethograms in order to visualize and quantify observed courtship behaviour

A Silent-Speech Interface using Electro-Optical Stomatography

Sprachtechnologie ist eine große und wachsende Industrie, die das Leben von technologieinteressierten Nutzern auf zahlreichen Wegen bereichert. Viele potenzielle Nutzer werden jedoch ausgeschlossen: Nämlich alle Sprecher, die nur schwer oder sogar gar nicht Sprache produzieren können. Silent-Speech Interfaces bieten einen Weg, mit Maschinen durch ein bequemes sprachgesteuertes Interface zu kommunizieren ohne dafür akustische Sprache zu benötigen. Sie können außerdem prinzipiell eine Ersatzstimme stellen, indem sie die intendierten Äußerungen, die der Nutzer nur still artikuliert, künstlich synthetisieren. Diese Dissertation stellt ein neues Silent-Speech Interface vor, das auf einem neu entwickelten Messsystem namens Elektro-Optischer Stomatografie und einem neuartigen parametrischen Vokaltraktmodell basiert, das die Echtzeitsynthese von Sprache basierend auf den gemessenen Daten ermöglicht. Mit der Hardware wurden Studien zur Einzelworterkennung durchgeführt, die den Stand der Technik in der intra- und inter-individuellen Genauigkeit erreichten und übertrafen. Darüber hinaus wurde eine Studie abgeschlossen, in der die Hardware zur Steuerung des Vokaltraktmodells in einer direkten Artikulation-zu-Sprache-Synthese verwendet wurde. Während die Verständlichkeit der Synthese von Vokalen sehr hoch eingeschätzt wurde, ist die Verständlichkeit von Konsonanten und kontinuierlicher Sprache sehr schlecht. Vielversprechende Möglichkeiten zur Verbesserung des Systems werden im Ausblick diskutiert.:Statement of authorship iii Abstract v List of Figures vii List of Tables xi Acronyms xiii 1. Introduction 1 1.1. The concept of a Silent-Speech Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.2. Structure of this work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2. Fundamentals of phonetics 7 2.1. Components of the human speech production system . . . . . . . . . . . . . . . . . . . 7 2.2. Vowel sounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.3. Consonantal sounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.4. Acoustic properties of speech sounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.5. Coarticulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.6. Phonotactics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.7. Summary and implications for the design of a Silent-Speech Interface (SSI) . . . . . . . 21 3. Articulatory data acquisition techniques in Silent-Speech Interfaces 25 3.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 3.2. Scope of the literature review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 3.3. Video Recordings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 3.4. Ultrasonography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 3.5. Electromyography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 3.6. Permanent-Magnetic Articulography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 3.7. Electromagnetic Articulography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3.8. Radio waves . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 3.9. Palatography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 3.10.Conclusion and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 4. Electro-Optical Stomatography 55 4.1. Contact sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 4.2. Optical distance sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 4.3. Lip sensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 4.4. Sensor Unit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 4.5. Control Unit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 4.6. Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 5. Articulation-to-Text 99 5.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 5.2. Command word recognition pilot study . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 5.3. Command word recognition small-scale study . . . . . . . . . . . . . . . . . . . . . . . . 102 6. Articulation-to-Speech 109 6.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 6.2. Articulatory synthesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 6.3. The six point vocal tract model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 6.4. Objective evaluation of the vocal tract model . . . . . . . . . . . . . . . . . . . . . . . . 116 6.5. Perceptual evaluation of the vocal tract model . . . . . . . . . . . . . . . . . . . . . . . . 120 6.6. Direct synthesis using EOS to control the vocal tract model . . . . . . . . . . . . . . . . 125 6.7. Pitch and voicing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 7. Summary and outlook 145 7.1. Summary of the contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 7.2. Outlook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 A. Overview of the International Phonetic Alphabet 151 B. Mathematical proofs and derivations 153 B.1. Combinatoric calculations illustrating the reduction of possible syllables using phonotactics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 B.2. Signal Averaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 B.3. Effect of the contact sensor area on the conductance . . . . . . . . . . . . . . . . . . . . 155 B.4. Calculation of the forward current for the OP280V diode . . . . . . . . . . . . . . . . . . 155 C. Schematics and layouts 157 C.1. Schematics of the control unit. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 C.2. Layout of the control unit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 C.3. Bill of materials of the control unit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 C.4. Schematics of the sensor unit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 C.5. Layout of the sensor unit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 C.6. Bill of materials of the sensor unit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 D. Sensor unit assembly 169 E. Firmware flow and data protocol 177 F. Palate file format 181 G. Supplemental material regarding the vocal tract model 183 H. Articulation-to-Speech: Optimal hyperparameters 189 Bibliography 191Speech technology is a major and growing industry that enriches the lives of technologically-minded people in a number of ways. Many potential users are, however, excluded: Namely, all speakers who cannot easily or even at all produce speech. Silent-Speech Interfaces offer a way to communicate with a machine by a convenient speech recognition interface without the need for acoustic speech. They also can potentially provide a full replacement voice by synthesizing the intended utterances that are only silently articulated by the user. To that end, the speech movements need to be captured and mapped to either text or acoustic speech. This dissertation proposes a new Silent-Speech Interface based on a newly developed measurement technology called Electro-Optical Stomatography and a novel parametric vocal tract model to facilitate real-time speech synthesis based on the measured data. The hardware was used to conduct command word recognition studies reaching state-of-the-art intra- and inter-individual performance. Furthermore, a study on using the hardware to control the vocal tract model in a direct articulation-to-speech synthesis loop was also completed. While the intelligibility of synthesized vowels was high, the intelligibility of consonants and connected speech was quite poor. Promising ways to improve the system are discussed in the outlook.:Statement of authorship iii Abstract v List of Figures vii List of Tables xi Acronyms xiii 1. Introduction 1 1.1. The concept of a Silent-Speech Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.2. Structure of this work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2. Fundamentals of phonetics 7 2.1. Components of the human speech production system . . . . . . . . . . . . . . . . . . . 7 2.2. Vowel sounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.3. Consonantal sounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.4. Acoustic properties of speech sounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.5. Coarticulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.6. Phonotactics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.7. Summary and implications for the design of a Silent-Speech Interface (SSI) . . . . . . . 21 3. Articulatory data acquisition techniques in Silent-Speech Interfaces 25 3.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 3.2. Scope of the literature review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 3.3. Video Recordings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 3.4. Ultrasonography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 3.5. Electromyography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 3.6. Permanent-Magnetic Articulography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 3.7. Electromagnetic Articulography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3.8. Radio waves . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 3.9. Palatography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 3.10.Conclusion and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 4. Electro-Optical Stomatography 55 4.1. Contact sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 4.2. Optical distance sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 4.3. Lip sensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 4.4. Sensor Unit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 4.5. Control Unit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 4.6. Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 5. Articulation-to-Text 99 5.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 5.2. Command word recognition pilot study . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 5.3. Command word recognition small-scale study . . . . . . . . . . . . . . . . . . . . . . . . 102 6. Articulation-to-Speech 109 6.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 6.2. Articulatory synthesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 6.3. The six point vocal tract model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 6.4. Objective evaluation of the vocal tract model . . . . . . . . . . . . . . . . . . . . . . . . 116 6.5. Perceptual evaluation of the vocal tract model . . . . . . . . . . . . . . . . . . . . . . . . 120 6.6. Direct synthesis using EOS to control the vocal tract model . . . . . . . . . . . . . . . . 125 6.7. Pitch and voicing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 7. Summary and outlook 145 7.1. Summary of the contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 7.2. Outlook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 A. Overview of the International Phonetic Alphabet 151 B. Mathematical proofs and derivations 153 B.1. Combinatoric calculations illustrating the reduction of possible syllables using phonotactics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 B.2. Signal Averaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 B.3. Effect of the contact sensor area on the conductance . . . . . . . . . . . . . . . . . . . . 155 B.4. Calculation of the forward current for the OP280V diode . . . . . . . . . . . . . . . . . . 155 C. Schematics and layouts 157 C.1. Schematics of the control unit. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 C.2. Layout of the control unit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 C.3. Bill of materials of the control unit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 C.4. Schematics of the sensor unit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 C.5. Layout of the sensor unit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 C.6. Bill of materials of the sensor unit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 D. Sensor unit assembly 169 E. Firmware flow and data protocol 177 F. Palate file format 181 G. Supplemental material regarding the vocal tract model 183 H. Articulation-to-Speech: Optimal hyperparameters 189 Bibliography 19

A Guideline for Environmental Games (GEG) and a randomized controlled evaluation of a game to increase environmental knowledge related to human population growth

People often have very little knowledge about the impact of unsustainable human population growth on the environment and social well-being especially in developing countries. Therefore, an efficient method should be explored in order to educate, and if possible, to convince the members of the public to realize the environmental and social problems caused by the unsustainable population growth. Digital Game-Based Learning (DGBL) has been highlighted by some studies as an innovative tool for learning enhancement. While only a handful of studies have scientifically evaluated the impact of DGBL on knowledge outcomes, the approach is an attractive tool to increase knowledge and motivate engagement with environmental issues surrounding population growth because of its potential to improve learners’ motivation and engagement thereby compared to traditional learning approaches. Therefore, the three primary research questions for this study are: 1) "Can a single-player digital game be an appropriate and attractive learning application for the players to gain insight about the relationship between the growing human population and the environmental issues?" 2) "How can we design environmental games for the players to gain insights about the relationship between the growing human population and the environmental issues via playing a game?" and 3) "What are the obstacles preventing the players from adapting environmental knowledge obtained from the learning mediums into the real-life?" To inform the development of an efficacious DGBL game to impact learning outcomes, critical reviews of environmental issues related to population growth as well as critical reviews of commercial and serious environmental games in terms of their educational and motivational values were undertaken in this study. The results of these critical reviews informed the development of a Guideline for Environmental Games (or GEG). The GEG was developed by combining the engaging game technology with environmental learning and persuasion theories. The GEG was then used to inform the development of a prototype game called THE GROWTH; a single-player, quiz-based, city-management game targeting young adolescents and adults. Multiple evaluation methods of the game were used to answer the three key research questions mentioned earlier. These methods included: 1) The Randomized Controlled Trial approach (RCT) where the participants were systematically divided into the experimental and the control group respectively and their knowledge scores (quantitative data) compared and analyzed, 2) The participants’ abilities to recall and describe the environmental and well-being issues were collected and analyzed qualitatively using The Content Analysis method (CA) and, 3) The participants’ overall feedback on the learning mediums was collected and analyzed to evaluate the motivational values of THE GROWTH itself. To this end, THE GROWTH was evaluated with 82 Thai-nationality participants (70 males and 12 females). The results showed that participants assigned to play THE GROWTH demonstrated greater environmental and social-well-being knowledge related to population growth (F(1,40) = 43.86, p = .006) compared to the control group participants assigned to a non-interactive reading activity (consistent with material presented in THE GROWTH). Furthermore, participants who played THE GROWTH recalled on average more content presented in the game when compared to participants who were presented with similar content in the reading material (t (59) = 3.35, p = .001). In terms of level of engagement, the study suggested that participants assigned to the game were more engaging with their learning medium on average when compared to participants assigned to the non-interactive reading activity. This is evidenced by the longer time participants spent on the task, the activity observed from participants’ recorded gameplay, and their positive responses in the survey. The semi-structured interviews used in this study highlighted the participants’ attitudes towards the environmental, social, and technological issues. Although the participants’ perceived behavioural intention towards the environmental commitments were not statistically differed between the two study group, their responses still provide some evidences that leaps may occur from the learning mediums to the real-world context. Furthermore, these responses can be valuable evidences for the policy makers and for the future development of environmental serious games. Overall, the results suggested that digital environmental games such as THE GROWTH might be an effective and motivational tool in promote the learning about sustainable population size, the environment, and the social well-being. The game’s ability to convince the participants to change towards sustainable lifestyles, however, might be subjected to the future research and other real-world circumstances such as the governmental and public supports. In summary, the research in this thesis makes the following contributions to knowledge: • The Guideline for Environmental Games (GEG) contributes to knowledge about making theoretically-based environmental games. It has particular significance because the guideline was validated by demonstrating learning improvements in a systematic randomized controlled trial. • The use of Multi-Strategy Study Design where multiple systematic evaluation methods were used in conjunction to provide conclusive findings about the efficacy of DGBL to impact outcomes. • THE GROWTH itself is a contribution to applied research as an example of an effective DGBL learning tool

Learning Internal State Memory Representations from Observation

Learning from Observation (LfO) is a machine learning paradigm that mimics how people learn in daily life: learning how to do something simply by watching someone else do it. LfO has been used in various applications, from video game agent creation to driving a car, but it has always been limited by the inability of an observer to know what a performing entity chooses to remember as they act in an environment. Various methods have either ignored the effects of memory or otherwise made simplistic assumptions about its structure. In this dissertation, we propose a new method, Memory Composition Learning, that captures the influence of a performer\u27s memory in an observed behavior through the creation of an auxiliary memory feature set that explicitly models the aspects of the environment with significance for future decisions, and which can be used with a machine learning technique to provide salient information from memory. It advances the state of the art by automatically learning the internal structure of memory instead of ignoring or predefining it. This research is difficult in that memory modeling is an unsupervised learning problem that we elect to solve solely from unobtrusive observation. This research is significant for LfO in that it will allow learning techniques that otherwise could not use information from memory to use a tailored set of learned memory features that capture salient influences from memory and enable decision-making based on these influences for more effective learning performance. To validate our hypothesis, we implemented a prototype for modeling observed memory influences with our approach and applied it to simulated vacuum cleaner and lawn mower domains. Our investigation revealed that MCL was able to automatically learn memory features that describe the influences on an observed actor\u27s internal state, and which improved learning performance of observed behaviors

LIPIcs, Volume 244, ESA 2022, Complete Volume

LIPIcs, Volume 244, ESA 2022, Complete Volum