Learning to Map Natural Language to Executable Programs Over Databases

Natural language is a fundamental form of information and communication and is becoming the next frontier in computer interfaces. As the amount of data available online has increased exponentially, so has the need for Natural Language Interfaces (NLIs, which is not used for natural language inference in this thesis) to connect the data and the user by easily using natural language, significantly promoting the possibility and efficiency of information access for many users besides data experts. All consumer-facing software will one day have a dialogue interface, and this is the next vital leap in the evolution of search engines. Such intelligent dialogue systems should understand the meaning of language grounded in various contexts and generate effective language responses in different forms for information requests and human-computer communication.Developing these intelligent systems is challenging due to (1) limited benchmarks to drive advancements, (2) alignment mismatches between natural language and formal programs, (3) lack of trustworthiness and interpretability, (4) context dependencies in both human conversational interactions and the target programs, and (5) joint language understanding between dialog questions and NLI environments (e.g. databases and knowledge graphs). This dissertation presents several datasets, neural algorithms, and language models to address these challenges for developing deep learning technologies for conversational natural language interfaces (more specifically, NLIs to Databases or NLIDB). First, to drive advancements towards neural-based conversational NLIs, we design and propose several complex and cross-domain NLI benchmarks, along with introducing several datasets. These datasets enable training large, deep learning models. The evaluation is done on unseen databases. (e.g., about course arrangement). Systems must generalize well to not only new SQL queries but also to unseen database schemas to perform well on these tasks. Furthermore, in real-world applications, users often access information in a multi-turn interaction with the system by asking a sequence of related questions. The users may explicitly refer to or omit previously mentioned entities and constraints and may introduce refinements, additions, or substitutions to what has already been said. Therefore, some of them require systems to model dialog dynamics and generate natural language explanations for user verification. The full dialogue interaction with the system’s responses is also important as this supports clarifying ambiguous questions, verifying returned results, and notifying users of unanswerable or unrelated questions. A robust dialogue-based NLI system that can engage with users by forming its responses has thus become an increasingly necessary component for the query process. Moreover, this thesis presents the development of scalable algorithms designed to parse complex and sequential questions to formal programs (e.g., mapping questions to SQL queries that can execute against databases). We propose a novel neural model that utilizes type information from knowledge graphs to better understand rare entities and numbers in natural language questions. We also introduce a neural model based on syntax tree neural networks, which was the first methodology proposed for generating complex programs from language. Finally, language modeling creates contextualized vector representations of words by training a model to predict the next word given context words, which are the basis of deep learning for NLP. Recently, pre-trained language models such as BERT and RoBERTa achieve tremendous success in many natural language processing tasks such as text understanding and reading comprehension. However, most language models are pre-trained only on free-text such as Wikipedia articles and Books. Given that language in semantic parsing is usually related to some formal representations such as logic forms and SQL queries and has to be grounded in structural environments (e.g., databases), we propose better language models for NLIs by enforcing such compositional interpolation in them. To show they could better jointly understand dialog questions and NLI environments (e.g. databases and knowledge graphs), we show that these language models achieve new state-of-the-art results for seven representative tasks on semantic parsing, dialogue state tracking, and question answering. Also, our proposed pre-training method is much more effective than other prior work

An Approach for Intention-Driven, Dialogue-Based Web Search

Web search engines facilitate the achievement of Web-mediated tasks, including information retrieval, Web page navigation, and online transactions. These tasks often involve goals that pertain to multiple topics, or domains. Current search engines are not suitable for satisfying complex, multi-domain needs due to their lack of interactivity and knowledge. This thesis presents a novel intention-driven, dialogue-based Web search approach that uncovers and combines users\u27 multi-domain goals to provide helpful virtual assistance. The intention discovery procedure uses a hierarchy of Partially Observable Markov Decision Process-based dialogue managers and a backing knowledge base to systematically explore the dialogue\u27s information space, probabilistically refining the perception of user goals. The search approach has been implemented in IDS, a search engine for online gift shopping. A usability study comparing IDS-based searching with Google-based searching found that the IDS-based approach takes significantly less time and effort, and results in higher user confidence in the retrieved results

Web knowledge bases

Knowledge is key to natural language understanding. References to specific people, places and things in text are crucial to resolving ambiguity and extracting meaning. Knowledge Bases (KBs) codify this information for automated systems — enabling applications such as entity-based search and question answering. This thesis explores the idea that sites on the web may act as a KB, even if that is not their primary intent. Dedicated kbs like Wikipedia are a rich source of entity information, but are built and maintained at an ongoing cost in human effort. As a result, they are generally limited in terms of the breadth and depth of knowledge they index about entities. Web knowledge bases offer a distributed solution to the problem of aggregating entity knowledge. Social networks aggregate content about people, news sites describe events with tags for organizations and locations, and a diverse assortment of web directories aggregate statistics and summaries for long-tail entities notable within niche movie, musical and sporting domains. We aim to develop the potential of these resources for both web-centric entity Information Extraction (IE) and structured KB population. We first investigate the problem of Named Entity Linking (NEL), where systems must resolve ambiguous mentions of entities in text to their corresponding node in a structured KB. We demonstrate that entity disambiguation models derived from inbound web links to Wikipedia are able to complement and in some cases completely replace the role of resources typically derived from the KB. Building on this work, we observe that any page on the web which reliably disambiguates inbound web links may act as an aggregation point for entity knowledge. To uncover these resources, we formalize the task of Web Knowledge Base Discovery (KBD) and develop a system to automatically infer the existence of KB-like endpoints on the web. While extending our framework to multiple KBs increases the breadth of available entity knowledge, we must still consolidate references to the same entity across different web KBs. We investigate this task of Cross-KB Coreference Resolution (KB-Coref) and develop models for efficiently clustering coreferent endpoints across web-scale document collections. Finally, assessing the gap between unstructured web knowledge resources and those of a typical KB, we develop a neural machine translation approach which transforms entity knowledge between unstructured textual mentions and traditional KB structures. The web has great potential as a source of entity knowledge. In this thesis we aim to first discover, distill and finally transform this knowledge into forms which will ultimately be useful in downstream language understanding tasks

Designing Embodied Interactive Software Agents for E-Learning: Principles, Components, and Roles

Embodied interactive software agents are complex autonomous, adaptive, and social software systems with a digital embodiment that enables them to act on and react to other entities (users, objects, and other agents) in their environment through bodily actions, which include the use of verbal and non-verbal communicative behaviors in face-to-face interactions with the user. These agents have been developed for various roles in different application domains, in which they perform tasks that have been assigned to them by their developers or delegated to them by their users or by other agents. In computer-assisted learning, embodied interactive pedagogical software agents have the general task to promote human learning by working with students (and other agents) in computer-based learning environments, among them e-learning platforms based on Internet technologies, such as the Virtual Linguistics Campus (www.linguistics-online.com). In these environments, pedagogical agents provide contextualized, qualified, personalized, and timely assistance, cooperation, instruction, motivation, and services for both individual learners and groups of learners. This thesis develops a comprehensive, multidisciplinary, and user-oriented view of the design of embodied interactive pedagogical software agents, which integrates theoretical and practical insights from various academic and other fields. The research intends to contribute to the scientific understanding of issues, methods, theories, and technologies that are involved in the design, implementation, and evaluation of embodied interactive software agents for different roles in e-learning and other areas. For developers, the thesis provides sixteen basic principles (Added Value, Perceptible Qualities, Balanced Design, Coherence, Consistency, Completeness, Comprehensibility, Individuality, Variability, Communicative Ability, Modularity, Teamwork, Participatory Design, Role Awareness, Cultural Awareness, and Relationship Building) plus a large number of specific guidelines for the design of embodied interactive software agents and their components. Furthermore, it offers critical reviews of theories, concepts, approaches, and technologies from different areas and disciplines that are relevant to agent design. Finally, it discusses three pedagogical agent roles (virtual native speaker, coach, and peer) in the scenario of the linguistic fieldwork classes on the Virtual Linguistics Campus and presents detailed considerations for the design of an agent for one of these roles (the virtual native speaker)

An analysis of the application of AI to the development of intelligent aids for flight crew tasks

This report presents the results of a study aimed at developing a basis for applying artificial intelligence to the flight deck environment of commercial transport aircraft. In particular, the study was comprised of four tasks: (1) analysis of flight crew tasks, (2) survey of the state-of-the-art of relevant artificial intelligence areas, (3) identification of human factors issues relevant to intelligent cockpit aids, and (4) identification of artificial intelligence areas requiring further research

Ferocious Logics: Unmaking the Algorithm

Contemporary power manifests in the algorithmic. And yet this power seems incomprehensible: understood as code, it becomes apolitical; understood as a totality, it becomes overwhelming. This book takes an alternate approach, using it to unravel the operations of Uber and Palantir, Airbnb and Amazon Alexa. Moving off the whiteboard and into the world, the algorithmic must negotiate with frictions - the 'merely' technical routines of distributing data and running tasks coming together into broader social forces that shape subjectivities, steer bodies, and calibrate relationships. Driven by the imperatives of capital, the algorithmic exhausts subjects and spaces, a double move seeking to both exhaustively apprehend them and exhaust away their productivities. But these on-the-ground encounters also reveal that force is never guaranteed. The irreducibility of the world renders logic inadequate and control gives way to contingency

Comprehension based adaptive learning systems

Conversational Intelligent Tutoring Systems aim to mimic the adaptive behaviour of human tutors by delivering tutorial content as part of a dynamic exchange of information conducted using natural language. Deciding when it is beneficial to intervene in a student’s learning process is an important skill for tutoring. Human tutors use prior knowledge about the student, discourse content and learner non-verbal behaviour to choose when intervention will help learners overcome impasse. Experienced human tutors adapt discourse and pedagogy based on recognition of comprehension and non-comprehension indicative learner behaviour. In this research non-verbal behaviour is explored as a method of computationally analysing reading comprehension so as to equip an intelligent conversational agent with the human-like ability to estimate comprehension from non-verbal behaviour as a decision making trigger for feedback, prompts or hints. This thesis presents research that combines a conversational intelligent tutoring system (CITS) with near real-time comprehension classification based on modelling of e-learner non-verbal behaviour to estimate learner comprehension during on-screen conversational tutoring and to use comprehension classifications as a trigger for intervening with hints, prompts or feedback for the learner. To improve the effectiveness of tuition in e-learning, this research aims to design, develop and demonstrate novel computational methods for modelling e-learner comprehension of on-screen information in near real-time and for adapting CITS tutorial discourse and pedagogy in response to perception of comprehension indicative behaviour. The contribution of this research is to detail the motivation for, design of, and evaluation of a system which has the human-like ability to introduce micro-adaptive feedback into tutorial discourse in response to automatic perception of e-learner reading comprehension. This research evaluates empirically whether e-learner non-verbal behaviour can be modelled to classify comprehension in near real-time and presents a near real-time comprehension classification system which achieves normalised comprehension classification accuracy of 75%. Understanding e-learner comprehension creates exciting opportunities for advanced personalisation of materials, discourse, challenge and the digital environment itself. The research suggests a benefit is gained from comprehension based adaptation in conversational intelligent tutoring systems, with a controlled trial of a comprehension based adaptive CITS called Hendrix 2.0 showing increases in tutorial assessment scores of up to 17% when comprehension based discourse adaptation is deployed to scaffold the learning experience

Approaching algorithmic power

Contemporary power manifests in the algorithmic. Emerging quite recently as an object of study within media and communications, cultural research, gender and race studies, and urban geography, the algorithm often seems ungraspable. Framed as code, it becomes proprietary property, black-boxed and inaccessible. Framed as a totality, its becomes overwhelmingly complex, incomprehensible in its operations. Framed as a procedure, it becomes a technique to be optimised, bracketing out the political. In struggling to adequately grasp the algorithmic as an object of study, to unravel its mechanisms and materialities, these framings offer limited insight into how algorithmic power is initiated and maintained. This thesis instead argues for an alternative approach: firstly, that the algorithmic is coordinated by a coherent internal logic, a knowledge-structure that understands the world in particular ways; second, that the algorithmic is enacted through control, a material and therefore observable performance which purposively influences people and things towards a predetermined outcome; and third, that this complex totality of architectures and operations can be productively analysed as strategic sociotechnical clusters of machines. This method of inquiry is developed with and tested against four contemporary examples: Uber, Airbnb, Amazon Alexa, and Palantir Gotham. Highly profitable, widely adopted and globally operational, they exemplify the algorithmic shift from whiteboard to world. But if the world is productive, it is also precarious, consisting of frictional spaces and antagonistic subjects. Force cannot be assumed as unilinear, but is incessantly negotiated—operations of parsing data and processing tasks forming broader operations that strive to establish subjectivities and shape relations. These negotiations can fail, destabilised by inadequate logics and weak control. A more generic understanding of logic and control enables a historiography of the algorithmic. The ability to index information, to structure the flow of labor, to exert force over subjects and spaces— these did not emerge with the microchip and the mainframe, but are part of a longer lineage of calculation. Two moments from this lineage are examined: house-numbering in the Habsburg Empire and punch-card machines in the Third Reich. Rather than revolutionary, this genealogy suggests an evolutionary process, albeit uneven, linking the computation of past and present. The thesis makes a methodological contribution to the nascent field of algorithmic studies. But more importantly, it renders algorithmic power more intelligible as a material force. Structured and implemented in particular ways, the design of logic and control construct different versions, or modalities, of algorithmic power. This power is political, it calibrates subjectivities towards certain ends, it prioritises space in specific ways, and it privileges particular practices whilst suppressing others. In apprehending operational logics, the practice of method thus foregrounds the sociopolitical dimensions of algorithmic power. As the algorithmic increasingly infiltrates into and governs the everyday, the ability to understand, critique, and intervene in this new field of power becomes more urgent

Ferocious Logics

Contemporary power manifests in the algorithmic. And yet this power seems incomprehensible: understood as code, it becomes apolitical; understood as a totality, it becomes overwhelming. This book takes an alternate approach, using it to unravel the operations of Uber and Palantir, Airbnb and Amazon Alexa. Moving off the whiteboard and into the world, the algorithmic must negotiate with frictions—the ‘merely’ technical routines of distributing data and running tasks coming together into broader social forces that shape subjectivities, steer bodies, and calibrate relationships. Driven by the imperatives of capital, the algorithmic exhausts subjects and spaces, a double move seeking to both exhaustively apprehend them and exhaust away their productivities. But these on-the-ground encounters also reveal that force is never guaranteed. The irreducibility of the world renders logic inadequate and control gives way to contingency

Estado del arte para la elaboración de pruebas utilizando el procesamiento de lenguaje natural en el área de física y matemática

El presente estado del arte analiza la investigación actual sobre el uso del procesamiento natural de lenguaje (por sus siglas NLP en inglés) en la creación de pruebas en materias a nivel universitario de física y matemáticas. Los resultados muestran que los investigadores están conscientes de la necesidad de automatizar la creación de pruebas en los sistemas de aprendizaje electrónico y ven el aprendizaje automático como una solución prometedora. Los hallazgos de este análisis del estado del arte brindan información sobre la investigación en el campo de la generación de pruebas en física y matemáticas. Estos descubrimientos complementan la información actual y sirven como base para investigaciones y progreso en el campo de la evaluación educativa.These state-of-the-art reviews current research on the use of natural language processing (by its acronym NLP in English) in the creation of tests in college-level subjects of physics and mathematics. The results show that researchers are aware of the need to automate test creation in e-learning systems and see machine learning as a promising solution. The findings of this state-of-the-art analysis provide insights into research in the field of evidence generation in physics and mathematics. These findings complement current information and serve as the basis for research and progress in the field of educational assessment