An agile model-driven method for involving end-users in DSL development

[EN] Domain-specific languages (DSLs) are considered to be a powerful tool for enhancing the efficiency of software developers and bring software development closer to end-users from complex domains. However, the successful development of a DSL for a complex domain is a challenge from the technical point of view and because end-user acceptance is key. Despite this fact, the relevant role of end-users during DSL development has traditionally been neglected. Normally, end-users participate at the beginning to communicate their preferences but they do not participate again until the DSL is completely implemented. As a consequence, if the language to develop reaches a complex domain, the chances that errors appear in the DSL are higher and solving them could involve large modifications that could have been avoided. As a solution, in this PhD thesis, we propose an agile, model-driven method to involve end-users in DSL development. This thesis researches if the combination of best practices from the model-driven development (MDD) discipline and best practices from agile methods is a suitable approach to involve end-users in the DSL development process. In order to validate the proposal, we have selected a highly complex domain such as the genetic analysis domain and we have collaborated with geneticists from three organizations. The proposed method has been used to involve these geneticists in the development of a DSL for the creation of genetic analysis pipelines. Simultaneously, we have carried out an empirical experiment to validate whether end-users and developers were satisfied with the proposal.[ES] Los lenguajes específicos de dominio (DSLs) son una herramienta muy potente para mejorar la eficiencia de los desarrolladores de software, así como para acercar el desarrollo software a usuarios sin conocimientos informáticos. Sin embargo, su principal problema es que desarrollar un DSL es complejo; no sólo desde el punto de vista técnico, sino especialmente porque la aceptación de dicho lenguaje por parte de los usuarios finales es clave. A pesar de este hecho, los métodos tradicionales de desarrollo de DSLs no enfatizan el importante rol de los usuarios finales durante el desarrollo. Normalmente, los usuarios participan al inicio para comunicar sus preferencias, pero no vuelven a participar hasta que el DSL está completamente desarrollado. Si el lenguaje a desarrollar aborda un dominio complejo, la posibilidad de que existan errores en el DSL es mayor, y su solución podría conllevar a modificaciones de gran calibre que podrían haberse evitado. Como solución, en esta tesis proponemos un método de desarrollo de DSLs, ágil, y dirigido por modelos que involucra a los usuarios finales. Esta tesis investiga si la combinación de buenas prácticas del desarrollo dirigido por modelos (MDD) y de buenas prácticas de métodos ágiles es adecuada para involucrar a los usuarios finales en el desarrollo de DSLs. Para validar la idoneidad de la propuesta, se ha seleccionado un dominio complejo como el de los análisis genéticos y se ha colaborado con un conjunto de genetistas procedentes de tres organizaciones. El método propuesto se ha utilizado para involucrar a dichos genetistas en el desarrollo de un DSL para la creación de pipelines para el análisis genético. Conjuntamente, se ha llevado a cabo un experimento empírico para validar si los usuarios finales y los desarrolladores están satisfechos con la propuesta de la presente tesis. En resumen, las contribuciones principales de esta tesis doctoral son el diseño e implementación de un método innovador, ágil y dirigido por modelos para involucrar a los usuarios finales en el desarrollo de DSLs, así como la validación de dicha propuesta en un entorno industrial en un desarrollo real de un DSL.[CA] Els llenguatges específics de domini (DSLs) son una ferramenta molt potent per a millorar l'eficiència dels desenvolupadors de programari, així com per a apropar el desenvolupament de programari a usuaris sense coneixements informàtics. El problema es que desenvolupar un DSL es complex, no sols des del punt de vista tècnic, sinó especialment perquè l'acceptació de dit llenguatge per part dels usuaris finals es clau. Malgrat aquest fet, els mètodes tradicionals de desenvolupament de DSLs no emfatitzen l'important rol dels usuaris finals durant el desenvolupament. Normalment, els usuaris participen a l'inici per a comunicar les seues preferències, però no tornen a participar fins que el DSL està completament desenvolupat. Si el llenguatge a desenvolupar aborda un domini complex, la possibilitat de que hi hagen errors en el DSL es major i solucionar-los podria implicar modificacions de gran calibre que podrien haver-se evitat. Com a solució, en aquesta tesis proposem un mètode de desenvolupament de DSLs, àgil i dirigit per models que involucra als usuaris finals. Aquesta tesis investiga si la combinació de bones pràctiques del desenvolupament dirigit per models (MDD) i de bones pràctiques de mètodes àgils es adequada per a involucrar els usuaris finals en el desenvolupament de DSLs. Per a validar la idoneïtat de la proposta, s'ha seleccionat un domini complex com el dels anàlisis genètics i s'ha col·laborat amb un conjunt de genetistes procedents de tres organitzacions. El mètode s'ha utilitzat per a involucrar a dits genetistes en el desenvolupament d'un DSL per a la creació de pipelines per al anàlisis genètic. Al mateix temps, s'ha dut a terme un experiment empíric per a validar si tant els usuaris finals com els desenvolupadors estan satisfets amb la proposta de la present tesis. En resum, les contribucions principals d'aquesta tesis doctoral son el disseny i implementació d'un mètode innovador, àgil i dirigit per models per a involucrar als usuaris finals en el desenvolupament de DSLs, així com la validació de la proposta en un entorn industrial amb un desenvolupament real d'un DSL.Villanueva Del Pozo, MJ. (2016). An agile model-driven method for involving end-users in DSL development [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/60156TESI

Solo: Data Discovery Using Natural Language Questions Via A Self-Supervised Approach

Most deployed data discovery systems, such as Google Datasets, and open data portals only support keyword search. Keyword search is geared towards general audiences but limits the types of queries the systems can answer. We propose a new system that lets users write natural language questions directly. A major barrier to using this learned data discovery system is it needs expensive-to-collect training data, thus limiting its utility. In this paper, we introduce a self-supervised approach to assemble training datasets and train learned discovery systems without human intervention. It requires addressing several challenges, including the design of self-supervised strategies for data discovery, table representation strategies to feed to the models, and relevance models that work well with the synthetically generated questions. We combine all the above contributions into a system, Solo, that solves the problem end to end. The evaluation results demonstrate the new techniques outperform state-of-the-art approaches on well-known benchmarks. All in all, the technique is a stepping stone towards building learned discovery systems. The code is open-sourced at https://github.com/TheDataStation/soloComment: To appear at Sigmod 202

Transfer Learning in Natural Language Processing through Interactive Feedback

Machine learning models cannot easily adapt to new domains and applications. This drawback becomes detrimental for natural language processing (NLP) because language is perpetually changing. Across disciplines and languages, there are noticeable differences in content, grammar, and vocabulary. To overcome these shifts, recent NLP breakthroughs focus on transfer learning. Through clever optimization and engineering, a model can successfully adapt to a new domain or task. However, these modifications are still computationally inefficient or resource-intensive. Compared to machines, humans are more capable at generalizing knowledge across different situations, especially in low-resource ones. Therefore, the research on transfer learning should carefully consider how the user interacts with the model. The goal of this dissertation is to investigate “human-in-the-loop” approaches for transfer learning in NLP. First, we design annotation frameworks for inductive transfer learning, which is the transfer of models across tasks. We create an interactive topic modeling system for users to find topics useful for classifying documents in multiple languages. The user-constructed topic model bridges improves classification accuracy and bridges cross-lingual gaps in knowledge. Next, we look at popular language models, like BERT, that can be applied to various tasks. While these models are useful, they still require a large amount of labeled data to learn a new task. To reduce labeling, we develop an active learning strategy which samples documents that surprise the language model. Users only need to annotate a small subset of these unexpected documents to adapt the language model for text classification. Then, we transition to user interaction in transductive transfer learning, which is the transfer of models across domains. We focus our efforts on low-resource languages to develop an interactive system for word embeddings. In this approach, the feedback from bilingual speakers refines the cross-lingual embedding space for classification tasks. Subsequently, we look at domain shift for tasks beyond text classification. Coreference resolution is fundamental for NLP applications, like question-answering and dialogue, but the models are typically trained and evaluated on one dataset. We use active learning to find spans of text in the new domain for users to label. Furthermore, we provide important insights on annotating spans for domain adaptation. Finally, we summarize the contributions of each chapter. We focus on aspects like the scope of applications and model complexity. We conclude with a discussion of future directions. Researchers may extend the ideas in our thesis to topics like user-centric active learning and proactive learning

Evaluating Machine Intelligence with Question Answering

Humans ask questions to learn about the world and to test knowledge understanding. The ability to ask questions combines aspects of intelligence unique to humans: language understanding, knowledge representation, and reasoning. Thus, building systems capable of intelligent question answering (QA) is a grand goal of natural language processing (NLP). To measure progress in NLP, we create "exams" for computer systems and compare their effectiveness against a reference point---often based on humans. How precisely we measure progress depends on whether we are building computer systems that optimize human satisfaction in information-seeking tasks or that measure progress towards intelligent QA. In the first part of this dissertation, we explore each goal in turn, how they differ, and describe their relationship to QA formats. As an example of an information-seeking evaluation, we introduce a new dialog QA task paired with a new evaluation method. Afterward, we turn our attention to using QA to evaluate machine intelligence. A good evaluation should be able to discriminate between lesser and more capable QA models. This dissertation explores three ways to improve the discriminative power of QA evaluations: (1) dynamic weighting of test questions, (2) a format that by construction tests multiple levels of knowledge, and (3) evaluation data that is created through human-computer collaboration. By dynamically weighting test questions, we challenge a foundational assumption of the de facto standard in QA evaluation---the leaderboard. Namely, we contend that contrary to nearly all QA and NLP evaluations which implicitly assign equal weights to examples by averaging scores, that examples are not equally useful for estimating machine (or human) QA ability. As any student may tell you, not all questions on an exam are equally difficult and in the worst-case questions are unsolvable. Drawing on decades of research in educational testing, we propose adopting an alternative evaluation methodology---Item Response Theory---that is widely used to score human exams (e.g., the SAT). By dynamically weighting questions, we show that this improves the reliability of leaderboards in discriminating between models of differing QA ability while also being helpful in the construction of new evaluation datasets. Having improved the scoring of models, we next turn to improving the format and data in QA evaluations. Our idea is simple. In most QA tasks (e.g., Jeopardy!), each question tests a single level of knowledge; in our task (the trivia game Quizbowl), we test multiple levels of knowledge with each question. Since each question tests multiple levels of knowledge, this decreases the likelihood that we learn nothing about the difference between two models (i.e., they are both correct or both wrong), which substantially increases discriminative power. Despite the improved format, we next show that while our QA models defeat accomplished trivia players, that they are overly reliant on brittle pattern matching, which indicates a failure to intelligently answer questions. To mitigate this problem, we introduce a new framework for building evaluation data where humans and machines cooperatively craft trivia questions that are difficult to answer through clever pattern matching tricks alone---while being no harder for humans. We conclude by sketching a broader vision for QA evaluation that combines the three components of evaluation we improve---scoring, format, and data---to create living evaluations and re-imagine the role of leaderboards