15,863 research outputs found
Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages
We present Samanantar, the largest publicly available parallel corpora
collection for Indic languages. The collection contains a total of 49.7 million
sentence pairs between English and 11 Indic languages (from two language
families). Specifically, we compile 12.4 million sentence pairs from existing,
publicly-available parallel corpora, and additionally mine 37.4 million
sentence pairs from the web, resulting in a 4x increase. We mine the parallel
sentences from the web by combining many corpora, tools, and methods: (a)
web-crawled monolingual corpora, (b) document OCR for extracting sentences from
scanned documents, (c) multilingual representation models for aligning
sentences, and (d) approximate nearest neighbor search for searching in a large
collection of sentences. Human evaluation of samples from the newly mined
corpora validate the high quality of the parallel sentences across 11
languages. Further, we extract 83.4 million sentence pairs between all 55 Indic
language pairs from the English-centric parallel corpus using English as the
pivot language. We trained multilingual NMT models spanning all these languages
on Samanantar, which outperform existing models and baselines on publicly
available benchmarks, such as FLORES, establishing the utility of Samanantar.
Our data and models are available publicly at
https://indicnlp.ai4bharat.org/samanantar/ and we hope they will help advance
research in NMT and multilingual NLP for Indic languages.Comment: Accepted to the Transactions of the Association for Computational
Linguistics (TACL
Enroller: an experiment in aggregating resources
This chapter describes a collaborative project between e-scientists and humanists working to create an online repository of linguistic data sets and tools. Corpora, dictionaries, and a thesaurus are brought together to enable a new method of research. It combines our most advanced knowledge in both computing and linguistic research techniques
Computer-based tracking, analysis, and visualization of linguistically significant nonmanual events in American Sign Language (ASL)
Our linguistically annotated American Sign Language (ASL) corpora have formed a basis for research to automate detection by
computer of essential linguistic information conveyed through facial expressions and head movements. We have tracked head position
and facial deformations, and used computational learning to discern specific grammatical markings. Our ability to detect, identify, and
temporally localize the occurrence of such markings in ASL videos has recently been improved by incorporation of (1) new techniques
for deformable model-based 3D tracking of head position and facial expressions, which provide significantly better tracking accuracy
and recover quickly from temporary loss of track due to occlusion; and (2) a computational learning approach incorporating 2-level
Conditional Random Fields (CRFs), suited to the multi-scale spatio-temporal characteristics of the data, which analyses not only
low-level appearance characteristics, but also the patterns that enable identification of significant gestural components, such as
periodic head movements and raised or lowered eyebrows. Here we summarize our linguistically motivated computational approach
and the results for detection and recognition of nonmanual grammatical markings; demonstrate our data visualizations, and discuss the
relevance for linguistic research; and describe work underway to enable such visualizations to be produced over large corpora and
shared publicly on the Web
Refining the use of the web (and web search) as a language teaching and learning resource
The web is a potentially useful corpus for language study because it provides examples of language that are contextualized and authentic, and is large and easily searchable. However, web contents are heterogeneous in the extreme, uncontrolled and hence 'dirty,' and exhibit features different from the written and spoken texts in other linguistic corpora. This article explores the use of the web and web search as a resource for language teaching and learning. We describe how a particular derived corpus containing a trillion word tokens in the form of n-grams has been filtered by word lists and syntactic constraints and used to create three digital library collections, linked with other corpora and the live web, that exploit the affordances of web text and mitigate some of its constraints
Meeting of the MINDS: an information retrieval research agenda
Since its inception in the late 1950s, the field of Information Retrieval (IR) has developed tools that help people find, organize, and analyze information. The key early influences on the field are well-known. Among them are H. P. Luhn's pioneering work, the development of the vector space retrieval model by Salton and his students, Cleverdon's development of the Cranfield experimental methodology, Spärck Jones' development of idf, and a series of probabilistic retrieval models by Robertson and Croft. Until the development of the WorldWideWeb (Web), IR was of greatest interest to professional information analysts such as librarians, intelligence analysts, the legal community, and the pharmaceutical industry
Challenges in development of the American Sign Language Lexicon Video Dataset (ASLLVD) corpus
The American Sign Language Lexicon Video Dataset (ASLLVD) consists of videos of >3,300 ASL signs in citation form, each produced by 1-6 native ASL signers, for a total of almost 9,800 tokens. This dataset, including multiple synchronized videos showing the signing from different angles, will be shared publicly once the linguistic annotations and verifications are complete. Linguistic annotations include gloss labels, sign start and end time codes, start and end handshape labels for both hands, morphological and articulatory classifications of sign type. For compound signs, the dataset includes annotations for each morpheme. To facilitate computer vision-based sign language recognition, the dataset also includes numeric ID labels for sign variants, video sequences in uncompressed-raw format, camera calibration sequences, and software for skin region extraction. We discuss here some of the challenges involved in the linguistic annotations and categorizations. We also report an example computer vision application that leverages the ASLLVD: the formulation employs a HandShapes Bayesian Network (HSBN), which models the transition probabilities between start and end handshapes in monomorphemic lexical signs. Further details and statistics for the ASLLVD dataset, as well as information about annotation conventions, are available from http://www.bu.edu/asllrp/lexicon
Exploration of biomedical knowledge for recurrent glioblastoma using natural language processing deep learning models
Abstract
Background
Efficient exploration of knowledge for the treatment of recurrent glioblastoma (GBM) is critical for both clinicians and researchers. However, due to the large number of clinical trials and published articles, searching for this knowledge is very labor-intensive. In the current study, using natural language processing (NLP), we analyzed medical research corpora related to recurrent glioblastoma to find potential targets and treatments.
Methods
We fine-tuned the SAPBERT, which was pretrained on biomedical ontologies, to perform question/answering (QA) and name entity recognition (NER) tasks for medical corpora. The model was fine-tuned with the SQUAD2 dataset and multiple NER datasets designed for QA task and NER task, respectively. Corpora were collected by searching the terms recurrent glioblastoma and drug target, published from 2000 to 2020 in the Web of science (N = 288 articles). Also, clinical trial corpora were collected from clinicaltrial.gov using the searching term of recurrent glioblastoma (N = 587 studies).
Results
For the QA task, the model showed an F1 score of 0.79. For the NER task, the model showed F1 scores of 0.90 and 0.76 for drug and gene name recognition, respectively. When asked what the molecular targets were promising for recurrent glioblastoma, the model answered that RTK inhibitors or LPA-1 antagonists were promising. From collected clinical trials, the model summarized them in the order of bevacizumab, temozolomide, lomustine, and nivolumab. Based on published articles, the model found the many drug-gene pairs with the NER task, and we presented them with a circus plot and related summarization (
https://github.com/bigwiz83/NLP_rGBM
).
Conclusion
Using NLP deep learning models, we could explore potential targets and treatments based on medical research and clinical trial corpora. The knowledge found by the models may be used for treating recurrent glioblastoma
- …