Clinfo.ai: An Open-Source Retrieval-Augmented Large Language Model System for Answering Medical Questions using Scientific Literature

The quickly-expanding nature of published medical literature makes it challenging for clinicians and researchers to keep up with and summarize recent, relevant findings in a timely manner. While several closed-source summarization tools based on large language models (LLMs) now exist, rigorous and systematic evaluations of their outputs are lacking. Furthermore, there is a paucity of high-quality datasets and appropriate benchmark tasks with which to evaluate these tools. We address these issues with four contributions: we release Clinfo.ai, an open-source WebApp that answers clinical questions based on dynamically retrieved scientific literature; we specify an information retrieval and abstractive summarization task to evaluate the performance of such retrieval-augmented LLM systems; we release a dataset of 200 questions and corresponding answers derived from published systematic reviews, which we name PubMed Retrieval and Synthesis (PubMedRS-200); and report benchmark results for Clinfo.ai and other publicly available OpenQA systems on PubMedRS-200.Comment: Preprint of an article published in Pacific Symposium on Biocomputing copyright 2024 World Scientific Publishing Co., Singapore, http://psb.stanford.edu

Apply deep learning to improve the question analysis model in the Vietnamese question answering system

Question answering (QA) system nowadays is quite popular for automated answering purposes, the meaning analysis of the question plays an important role, directly affecting the accuracy of the system. In this article, we propose an improvement for question-answering models by adding more specific question analysis steps, including contextual characteristic analysis, pos-tag analysis, and question-type analysis built on deep learning network architecture. Weights of extracted words through question analysis steps are combined with the best matching 25 (BM25) algorithm to find the best relevant paragraph of text and incorporated into the QA model to find the best and least noisy answer. The dataset for the question analysis step consists of 19,339 labeled questions covering a variety of topics. Results of the question analysis model are combined to train the question-answering model on the data set related to the learning regulations of Industrial University of Ho Chi Minh City. It includes 17,405 pairs of questions and answers for the training set and 1,600 pairs for the test set, where the robustly optimized BERT pre-training approach (RoBERTa) model has an F1-score accuracy of 74%. The model has improved significantly. For long and complex questions, the mode has extracted weights and correctly provided answers based on the question’s contents

Toward Stance-based Personas for Opinionated Dialogues

In the context of chit-chat dialogues it has been shown that endowing systems with a persona profile is important to produce more coherent and meaningful conversations. Still, the representation of such personas has thus far been limited to a fact-based representation (e.g. "I have two cats."). We argue that these representations remain superficial w.r.t. the complexity of human personality. In this work, we propose to make a step forward and investigate stance-based persona, trying to grasp more profound characteristics, such as opinions, values, and beliefs to drive language generation. To this end, we introduce a novel dataset allowing to explore different stance-based persona representations and their impact on claim generation, showing that they are able to grasp abstract and profound aspects of the author persona.Comment: Accepted at Findings of EMNLP 202

Weighting Passages Enhances Accuracy

We observe that in curated documents the distribution of the occurrences of salient terms, e.g., terms with a high Inverse Document Frequency, is not uniform, and such terms are primarily concentrated towards the beginning and the end of the document. Exploiting this observation, we propose a novel version of the classical BM25 weighting model, called BM25 Passage (BM25P), which scores query results by computing a linear combination of term statistics in the different portions of the document. We study a multiplicity of partitioning schemes of document content into passages and compute the collection-dependent weights associated with them on the basis of the distribution of occurrences of salient terms in documents. Moreover, we tune BM25P hyperparameters and investigate their impact on ad hoc document retrieval through fully reproducible experiments conducted using four publicly available datasets. Our findings demonstrate that our BM25P weighting model markedly and consistently outperforms BM25 in terms of effectiveness by up to 17.44% in NDCG@5 and 85% in NDCG@1, and up to 21% in MRR

Ранжирование документов при полнотекстовом поиске с учетом расстояния с использованием индексов с многокомпонентными ключами

The problem of proximity full-text search is considered. If a search query contains high-frequently occurring words, then multi-component key indexes deliver improvement of the search speed in comparison with ordinary inverted indexes. It was shown that we can increase the search speed up to 130 times in cases when queries consist of high-frequently occurring words. In this paper, we are investigating how the multi-component key indexes architecture affects the quality of the search. We consider several well-known methods of relevance ranking; these methods are of different authors. Using these methods we perform the search in the ordinary inverted index and then in the index that is enhanced with multi-component key indexes. The results show that with multi-component key indexes we obtain search results that are very near in terms of relevance ranking to the search results that are obtained by means of ordinary inverted indexes. © 2021 Udmurt State University. All rights reserved

Sistem Rekomendasi Dosen Pembimbing Berdasarkan Latar Belakang Menggunakan Metode Multi-Class Support Vector Machine Dan Weighted Product

Pada Fakultas Ilmu Komputer Universitas Brawijaya, pengerjaan skripsi dimulai dengan melakukan pembuatan praproposal yang berisi latar belakang dan bidang skripsi. Dalam pengerjaan skripsi, mahasiswa butuh pendampingan oleh dosen pembimbing. Dosen pembimbing berfungsi sebagai motivator, pendamping serta pemberi arahan bagi mahasiswa yang sedang mengerjakan skripsi. Dosen pembimbing menjadi krusial dalam pengerjaan skripsi seorang mahasiswa. Oleh karena hal tersebut, pemilihan dosen pembimbing yang memiliki bidang keahlian yang sesuai dengan topik skripsi sangat penting. Topik skripsi ditentukan dari latar belakang sebuah proposal skripsi. Untuk menjawab permasalahan dibutuhkan sebuah sistem yang mampu menentukan topik skripsi dan memprediksi dosen pembimbing yang sesuai berdasarkan latar belakang proposal skripsi. Rekomendasi dosen diberikan berdasarkan kesesuaian bidang dosen dengan topik dan beberapa data dosen yang didapatkan dari unit Pengelola Sistem Informasi, Infrastruktur TI dan Kehumasan Fakultas Ilmu Komputer (PSIK FILKOM) seperti jurusan dosen, sisa kuota bimbingan, tingkatan gelar, dan beban kerja. Hasil pengujian akurasi menghasilkan akurasi klasifikasi bidang skripsi sebesar 93,75% dan akurasi prediksi dosen pembimbing sebesar 57,14%. Hasil pengujian unit menunjukkan bahwa sistem 100% valid dan sesuai dengan kebutuhan. Hasil pengujian kompatibilitas menunjukkan bahwa sistem dapat digunakan di berbagai web browser, sistem operasi, dan platform. Hasil pengujian performa menunjukkan bahwa 85,71% bagian dari sistem telah responsive dan dapat digunakan dengan baik

Selective Query Processing: a Risk-Sensitive Selection of System Configurations

In information retrieval systems, search parameters are optimized to ensure high effectiveness based on a set of past searches and these optimized parameters are then used as the system configuration for all subsequent queries. A better approach, however, would be to adapt the parameters to fit the query at hand. Selective query expansion is one such an approach, in which the system decides automatically whether or not to expand the query, resulting in two possible system configurations. This approach was extended recently to include many other parameters, leading to many possible system configurations where the system automatically selects the best configuration on a per-query basis. To determine the ideal configurations to use on a per-query basis in real-world systems we developed a method in which a restricted number of possible configurations is pre-selected and then used in a meta-search engine that decides the best search configuration on a per query basis. We define a risk-sensitive approach for configuration pre-selection that considers the risk-reward trade-off between the number of configurations kept, and system effectiveness. For final configuration selection, the decision is based on query feature similarities. We find that a relatively small number of configurations (20) selected by our risk-sensitive model is sufficient to increase effectiveness by about 15% according(P@10, nDCG@10) when compared to traditional grid search using a single configuration and by about 20% when compared to learning to rank documents. Our risk-sensitive approach works for both diversity- and ad hoc-oriented searches. Moreover, the similarity-based selection method outperforms the more sophisticated approaches. Thus, we demonstrate the feasibility of developing per-query information retrieval systems, which will guide future research in this direction.Comment: 30 pages, 5 figures, 8 tables; submitted to TOIS ACM journa