Question Answering with distilled BERT models: A case study for Biomedical Data

In the healthcare industry today, 80% of data is unstructured (Razzak et al., 2019). The challenge this imposes on healthcare providers is that they rely on unstructured data to inform their decision-making. Although Electronic Health Records (EHRs) exist to integrate patient data, healthcare providers are still challenged with searching for information and answers contained within unstructured data. Prior NLP and Deep Learning research has shown that these methods can improve information extraction on unstructured medical documents. This research expands upon those studies by developing a Question Answering system using distilled BERT models. Healthcare providers can use this system on their local computers to search for and receive answers to specific questions about patients. This paper’s best TinyBERT and TinyBioBERT models had Mean Reciprocal Rank (MRRs) of 0.522 and 0.284 respectively. Based on these findings this paper concludes that TinyBERT performed better than TinyBioBERT on BioASQ task 9b data

Enhancing Biomedical Text Summarization and Question-Answering: On the Utility of Domain-Specific Pre-Training

Biomedical summarization requires large datasets to train for text generation. We show that while transfer learning offers a viable option for addressing this challenge, an in-domain pre-training does not always offer advantages in a BioASQ summarization task. We identify a suitable model architecture and use it to show a benefit of a general-domain pre-training followed by a task-specific fine-tuning in the context of a BioASQ summarization task, leading to a novel three-step fine-tuning approach that works with only a thousand in-domain examples. Our results indicate that a Large Language Model without domain-specific pre-training can have a significant edge in some domain-specific biomedical text generation tasks

Results of the seventh edition of the BioASQ Challenge

The results of the seventh edition of the BioASQ challenge are presented in this paper. The aim of the BioASQ challenge is the promotion of systems and methodologies through the organization of a challenge on the tasks of large-scale biomedical semantic indexing and question answering. In total, 30 teams with more than 100 systems participated in the challenge this year. As in previous years, the best systems were able to outperform the strong baselines. This suggests that state-of-the-art systems are continuously improving, pushing the frontier of research.Comment: 17 pages, 2 figure

Applying deep learning extreme multi-label classification to the biomedical and multilingual panoramas

Tese de mestrado em Bioinformática e Biologia Computacional, Universidade de Lisboa, Faculdade de Ciências, 2020A indexação automática de documentos é um passo fundamental para a organização de dados e para a extração de informação relevante dos mesmos. Esta extração de informação é realizada através de processos de prospecção de texto e de técnicas de processamento de linguagem natural que tornam a linguagem natural perceptível para o computador. Actualmente, muitas das soluções que são aplicadas a estes processos consistem em soluções de aprendizagem automática. No entanto, tem se assistido a um aumento contínuo da aplicação de soluções de aprendizagem profunda em tarefas de prospecção de texto e de processamento de linguagem natural visto que, graças aos desenvolvimentos contínuos ao longo dos últimos anos, estas soluções têm conseguido obter cada vez melhores resultados. Uma dessas técnicas é a classificação multi-rótulo extrema, uma técnica de processamento de linguagem natural que consiste na indexação de documentos com rótulos pertencentes a um conjunto que pode conter milhares ou mesmo milhões de possíveis rótulos. Este trabalho apresenta um sistema desenvolvido para as ciências biomédicas e para o domínio multilinguístico, através da adaptação de um algoritmo de classificação multi-rótulo extrema usando aprendizagem profunda. O sistema desenvolvido combina ainda um software de reconhecimento de entidades nomeadas com o algoritmo de classificação multi-rótulo extrema de forma a melhorar a atribuição de rótulos aos documentos biomédicos. Para testar o sistema desenvolvido, participei em três competições internacionais com foco na área das ciências biomédicas, nomeadamente na BioASQ task 8a, BioASQ task MESINESP e ainda na subtarefa CODING da competição CANTEMIST. O objectivo comum destas três competições consistia na indexação de documentos biomédicos com rótulos pertencentes a um dado vocabulário biomédico. No entanto, enquanto na task 8a os dados estavam escritos em Inglês, na task MESINESP e na CANTEMIST, os dados biomédicos estavam escritos em Espanhol. Nas competições da BioASQ, o sistema desenvolvido destacou-se sobretudo nas medidas de precisão, superando a grande maioria dos sistemas e ainda alcançando o 1º lugar por duas semanas consecutivas numa das medidas da BioASQ task 8a. Na subtarefa CODING da CANTEMIST, o sistema atingiu uma pontuação de 0.506 na medida mais relevante.Automatic document indexation is a fundamental step for data organization and information retrieval tasks. Information retrieval can be realized through processes of text mining and natural language processing techniques that make natural language understandable to the computer. Nowadays, most solutions that are applied to these processes use machine learning algorithms. However, thanks to continuous developments through recent years, there has been an increasing usage of deep learning solutions applied to text mining and natural language processing tasks, due to the continuous achievement of better results. One of those techniques is extreme multi-label classification, a natural language processing task consisting in the indexation of documents with labels from a label set that may contain thousands or even millions of possible labels. This work presents a system developed for the biomedical and multilingual panoramas based on the adaptation of a deep learning extreme multi-label classification algorithm. The developed system also combines a named entity recognition software with the extreme multi-label classification algorithm in order to improve the label classification of the biomedical documents. To test the developed system, I participated in three international challenges focused on the biomedical sciences, namely in the BioASQ task 8a, BioASQ task MESINESP and in CANTEMIST CODING subtask. The common goal of these three competitions was the indexation of biomedical documents with labels belonging to a specific biomedical vocabulary. However, while the data in task 8a was in English, in task MESINESP and in CANTEMIST the biomedical data was written in Spanish. In the BioASQ competitions, the system stood out in the precision measures, surpassing most competing systems and achieving the 1st place for two consecutive weeks in one evaluation measure in the BioASQ task 8a. In the CANTEMIST CODING subtask, the system achieved a score of 0.506 in the most relevant measure