Hyperparameter tuning for deep learning in natural language processing

Deep Neural Networks have advanced rapidly over the past several years. However, it still seems like a black art for many people to make use of them efficiently. The reason for this complexity is that obtaining a consistent and outstanding result from a deep architecture requires optimizing many parameters known as hyperparameters. Hyperparameter tuning is an essential task in deep learning, which can make significant changes in network performance. This paper is the essence of over 3000 GPU hours on optimizing a network for a text classification task on a wide array of hyperparameters. We provide a list of hyperparameters to tune in addition to their tuning impact on the network performance. The hope is that such a listing will provide the interested researchers a mean to prioritize their efforts and to modify their deep architecture for getting the best performance with the least effort

Towards Integration of Statistical Hypothesis Tests into Deep Neural Networks

We report our ongoing work about a new deep architecture working in tandem with a statistical test procedure for jointly training texts and their label descriptions for multi-label and multi-class classification tasks. A statistical hypothesis testing method is used to extract the most informative words for each given class. These words are used as a class description for more label-aware text classification. Intuition is to help the model to concentrate on more informative words rather than more frequent ones. The model leverages the use of label descriptions in addition to the input text to enhance text classification performance. Our method is entirely data-driven, has no dependency on other sources of information than the training data, and is adaptable to different classification problems by providing appropriate training data without major hyper-parameter tuning. We trained and tested our system on several publicly available datasets, where we managed to improve the state-of-the-art on one set with a high margin, and to obtain competitive results on all other ones.Comment: Accepted to ACL 201

Hybridní hluboké metody pro automatické odpovídání na otázky

Title: Hybrid Deep Question Answering Author: Ahmad Aghaebrahimian Institute: Institute of Formal and Applied Linguistics Supervisor: RNDr. Martin Holub, Ph.D., Institute of Formal and Applied Lin- guistics Abstract: As one of the oldest tasks of Natural Language Processing, Question Answering is one of the most exciting and challenging research areas with lots of scientific and commercial applications. Question Answering as a discipline in the conjunction of computer science, statistics, linguistics, and cognitive science is concerned with building systems that automatically retrieve answers to ques- tions posed by humans in a natural language. This doctoral dissertation presents the author's research carried out in this discipline. It highlights his studies and research toward a hybrid Question Answering system consisting of two engines for Question Answering over structured and unstructured data. The structured engine comprises a state-of-the-art Question Answering system based on knowl- edge graphs. The unstructured engine consists of a state-of-the-art sentence-level Question Answering system and a word-level Question Answering system with results near to human performance. This work introduces a new Question An- swering dataset for answering word- and sentence-level questions as well. Start- ing from a...Název práce: Hybridní hluboké metody pro automatické odpovídání na otázky Autor: Ahmad Aghaebrahimian Ústav: Ústav Formální a Aplikované Lingvistiky Vedoucí disertační práce: RNDr. Martin Holub, Ph.D., Ústav Formální a Ap- likované Lingvistiky Abstrakt: Automatické odpovídání na otázky jakožto jedna z nejstarších úloh z oblasti zpracování přirozeného jazyka je jednou z nejzajímavějších a nejná- ročnějších oblastí výzkumu s množstvím vědeckých a komerčních uplatnění. Od- povídání na otázky jakožto disciplína se ve spojení s informatikou, statistikou, lingvistikou a kognitivní vědou zabývá tvorbou systémů, které automaticky vy- hledávají odpovědi na otázky kladené lidmi v přirozeném jazyce. Tato doktorská disertační práce představuje autorův výzkum uskutečněný v uvedené oblasti. Au- tor předkládá především své studie a výzkum zaměřený na hybridní systémy pro odpovídání na otázky zahrnující vyhledávací stroje pracující jak se struktu- rovanými, tak s nestrukturovanými daty. Jádrem strukturovaného vyhledávacího stroje je state-of-the-art systém založený na znalostních grafech. Nestrukturovaný vyhledávací stroj je tvořen state-of-the-art systémem pro odpovídání na otázky na větné úrovni a systémem pro odpovídání na otázky na úrovni slov s výsledky, které se blíží tomu, čeho dosahují lidé. Tato práce představuje...Ústav formální a aplikované lingvistikyInstitute of Formal and Applied LinguisticsMatematicko-fyzikální fakultaFaculty of Mathematics and Physic

The Sentence End and Punctuation Prediction in NLG text (SEPP-NLG) shared task 2021

This paper describes the first Sentence End and Punctuation Prediction in Natural Language Generation (SEPP-NLG) shared task1 held at the SwissText conference 2021. The goal of the shared task was to develop solutions for the identification of sentence boundaries and the insertion of punctuation marks into texts produced by NLG systems. The data and submissions, and the codebase for the shared tasks are publicly available

Dynamic Assessment of Writing Skill in Advanced EFL Iranian Learners

AbstractDynamic Assessment (hereinafter DA) fundamentally is based on Vygotsky's theory of mediation and ZPD. In contrast to Traditional Assessment (hereinafter TA) which focuses on the product to show the current capability of learners, DA focuses on the process to predict their future achievement. This study intends to investigate the effectiveness of incorporating DA in improving teaching writing at advanced level among Iranian EFL learners. To fulfill this end twenty randomly chosen participants underwent a course of study in advanced writing for the period of two months and in eight sessions. All these participants received the same treatment, however, half of them, in the experimental group, were assessed dynamically and the other half, in the control group, were assessed traditionally. The participants’ essays in both groups were assessed by two distinct raters and their results were statistically analyzed. In order for the study results to be triangulated a questionnaire consisting of three questions was distributed among participants to support the quantitative results in a qualitative and subjective manner. The result of statistical analysis of the T-test which was used to distinguish between the experimental and the control group in addition to the questionnaires showed a significant change in the essays of the participants in the experimental group. The paper concludes with some practical implications for teachers, material developers and syllabus designers

Computational literature-based discovery for natural products research : current state and future prospects

Literature-based discovery (LBD) mines existing literature in order to generate new hypotheses by finding links between previously disconnected pieces of knowledge. Although automated LBD systems are becoming widespread and indispensable in a wide variety of knowledge domains, little has been done to introduce LBD to the field of natural products research. Despite growing knowledge in the natural product domain, most of the accumulated information is found in detached data pools. LBD can facilitate better contextualization and exploitation of this wealth of data, for example by formulating new hypotheses for natural product research, especially in the context of drug discovery and development. Moreover, automated LBD systems promise to accelerate the currently tedious and expensive process of lead identification, optimization, and development. Focusing on natural product research, we briefly reflect the development of automated LBD and summarize its methods and principal data sources. In a thorough review of published use cases of LBD in the biomedical domain, we highlight the immense potential of this data mining approach for natural product research, especially in context with drug discovery or repurposing, mode of action, as well as drug or substance interactions. Most of the 91 natural product-related discoveries in our sample of reported use cases of LBD were addressed at a computer science audience. Therefore, it is the wider goal of this review to introduce automated LBD to researchers who work with natural products and to facilitate the dialogue between this community and the developers of automated LBD systems

Hybrid Deep Question Answering

Title: Hybrid Deep Question Answering Author: Ahmad Aghaebrahimian Institute: Institute of Formal and Applied Linguistics Supervisor: RNDr. Martin Holub, Ph.D., Institute of Formal and Applied Lin- guistics Abstract: As one of the oldest tasks of Natural Language Processing, Question Answering is one of the most exciting and challenging research areas with lots of scientific and commercial applications. Question Answering as a discipline in the conjunction of computer science, statistics, linguistics, and cognitive science is concerned with building systems that automatically retrieve answers to ques- tions posed by humans in a natural language. This doctoral dissertation presents the author's research carried out in this discipline. It highlights his studies and research toward a hybrid Question Answering system consisting of two engines for Question Answering over structured and unstructured data. The structured engine comprises a state-of-the-art Question Answering system based on knowl- edge graphs. The unstructured engine consists of a state-of-the-art sentence-level Question Answering system and a word-level Question Answering system with results near to human performance. This work introduces a new Question An- swering dataset for answering word- and sentence-level questions as well. Start- ing from a..

Ontology-aware biomedical relation extraction

Automatically extracting relationships from biomedical texts among multiple sorts of entities is an essential task in biomedical natural language processing with numerous applications, such as drug development or repurposing, precision medicine, and other biomedical tasks requiring knowledge discovery. Current Relation Extraction systems mostly use one set of features, either as text, or more recently, as graph structures. The state-of-the-art systems often use resource-intensive hence slow algorithms and largely work for a particular type of relationship. However, a simple yet agile system that learns from different sets of features has the advantage of adaptability over different relationship types without an extra burden required for system re-design. We model RE as a classification task and propose a new multi-channel deep neural network designed to process textual and graph structures in separate input channels. We extend a Recurrent Neural Network with a Convolutional Neural Network to process three sets of features, namely, tokens, types, and graphs. We demonstrate that entity type and ontology graph structure provide better representations than simple token-based representations for Relation Extraction. We also experiment with various sources of knowledge, including data resources in the Unified Medical Language System to test our hypothesis. Extensive experiments on four well-studied biomedical benchmarks with different relationship types show that our system outperforms earlier ones. Thus, our system has state-of-the-art performance and allows processing millions of full-text scientific articles in a few days on one typical machine