6 research outputs found
URL-BERT: Training Webpage Representations via Social Media Engagements
Understanding and representing webpages is crucial to online social networks
where users may share and engage with URLs. Common language model (LM) encoders
such as BERT can be used to understand and represent the textual content of
webpages. However, these representations may not model thematic information of
web domains and URLs or accurately capture their appeal to social media users.
In this work, we introduce a new pre-training objective that can be used to
adapt LMs to understand URLs and webpages. Our proposed framework consists of
two steps: (1) scalable graph embeddings to learn shallow representations of
URLs based on user engagement on social media and (2) a contrastive objective
that aligns LM representations with the aforementioned graph-based
representation. We apply our framework to the multilingual version of BERT to
obtain the model URL-BERT. We experimentally demonstrate that our continued
pre-training approach improves webpage understanding on a variety of tasks and
Twitter internal and external benchmarks
Information retrieval and text mining technologies for chemistry
Efficient access to chemical information contained in scientific literature, patents, technical reports, or the web is a pressing need shared by researchers and patent attorneys from different chemical disciplines. Retrieval of important chemical information in most cases starts with finding relevant documents for a particular chemical compound or family. Targeted retrieval of chemical documents is closely connected to the automatic recognition of chemical entities in the text, which commonly involves the extraction of the entire list of chemicals mentioned in a document, including any associated information. In this Review, we provide a comprehensive and in-depth description of fundamental concepts, technical implementations, and current technologies for meeting these information demands. A strong focus is placed on community challenges addressing systems performance, more particularly CHEMDNER and CHEMDNER patents tasks of BioCreative IV and V, respectively. Considering the growing interest in the construction of automatically annotated chemical knowledge bases that integrate chemical information and biological data, cheminformatics approaches for mapping the extracted chemical names into chemical structures and their subsequent annotation together with text mining applications for linking chemistry with biological information are also presented. Finally, future trends and current challenges are highlighted as a roadmap proposal for research in this emerging field.A.V. and M.K. acknowledge funding from the European
Communityâs Horizon 2020 Program (project reference:
654021 - OpenMinted). M.K. additionally acknowledges the
Encomienda MINETAD-CNIO as part of the Plan for the
Advancement of Language Technology. O.R. and J.O. thank
the Foundation for Applied Medical Research (FIMA),
University of Navarra (Pamplona, Spain). This work was
partially funded by ConselleriÌa
de Cultura, EducacioÌn e OrdenacioÌn Universitaria (Xunta de Galicia), and FEDER (European Union), and the Portuguese Foundation for Science and Technology (FCT) under the scope of the strategic
funding of UID/BIO/04469/2013 unit and COMPETE 2020
(POCI-01-0145-FEDER-006684). We thank InÌigo GarciaÌ -Yoldi
for useful feedback and discussions during the preparation of
the manuscript.info:eu-repo/semantics/publishedVersio