Pre-trained biomedical language models for clinical NLP in Spanish

This work presents the first large-scale biomedical Spanish language models trained from scratch, using large biomedical corpora consisting of a total of 1.1B tokens and an EHR corpus of 95M tokens. We compared them against general-domain and other domain-specific models for Spanish on three clinical NER tasks. As main results, our models are superior across the NER tasks, rendering them more convenient for clinical NLP applications. Furthermore, our findings indicate that when enough data is available, pre-training from scratch is better than continual pre-training when tested on clinical tasks, raising an exciting research question about which approach is optimal. Our models and fine-tuning scripts are publicly available at HuggingFace and GitHub.This work was funded by the Spanish State Secretariat for Digitalization and Artificial Intelligence (SEDIA) within the framework of the Plan-TLPeer ReviewedPostprint (published version

El Instituto Ravetllat-Pla durante el franquismo (1939-1955): Estrategias comerciales y científicas del medicamento y la reconceptualización de la sueroterapia

El objetivo de esta memoria es contribuir a la comprensión histórica de la reconceptualización del medicamento. Nos referimos al fenómeno de transformación sufrido por aquellos fármacos que fueron pensados y producidos inicialmente para tratar una enfermedad concreta y que terminaron usándose en la terapéutica de otros procesos patológicos. Para el acercamiento que se propone, se ha utilizado el caso de estudio del Instituto Ravetllat-Pla entre 1939 y 1955 interesándonos por la producción, distribución y comercialización, fundamentalmente, de sus dos principales medicamentos: la «Hemo-antitoxina» y el «Suero Ravetllat-Pla». La historia de la medicina y de la farmacia han estudiado los medicamentos desde muy diversos ángulos y los han descrito como productos terapéuticos industrializados, como elaboraciones farmacéuticas reguladas, controladas y normalizadas y/o como artículos de consumo. Sin embargo, el estudio de la reconceptualización del medicamento necesita una integración de estas perspectivas de manera que la visión que nos ofrezca sea poliédrica. La correspondencia entre los agentes comerciales y el propio Instituto, nos han permitido analizar el debate que se produjo sobre el medicamento en el seno de su red comercial. La historia del Instituto Ravetllat-Pla resulta excepcional por la heterodoxia de sus directores. Este laboratorio se fundó en 1923 por el médico Ramón Pla Armengol (1880-1956) y el veterinario Joaquín Ravetllat Estech (1871-1923) para producir la Hemo-antitoxina y el Suero Ravetllat-Pla. El objetivo fundacional de este laboratorio fue legitimar la teoría Ravetllat-Pla a través de la fabricación y venta de los productos antituberculosos fundamentados en ella. En esta memoria estudiamos la trayectoria política de Ramón Pla hasta llegar a ser electo como diputado socialista a la Cortes en 1936 hasta su posterior exilio político. En esta primera parte analizamos su pensamiento heterodoxo y crítico con la “ciencia oficial” y también a su defensa de la seroterapia y la clínica en un momento que los antibióticos tomaban un mayor protagonismo en el debate médico. Por otro lado, mientras Ramón Pla se hallaba en el exilio, su hija Nuria Pla Monseny (1918-2011) partió a Burgos e ingresó en la Sección Femenina de la FET y de las JONS. En 1939, con la entrada de las tropas franquistas a Barcelona, Nuria Pla empezó a dirigir el Instituto Ravetllat-Pla que se encontraba en una situación científica, económica y comercial deplorable. La estrategia de Nuria Pla para lograr recuperar la actividad científica y comercial del Instituto Ravetllat-Pla comenzó por defender la propiedad del mismo, puesta en duda por el expediente abierto a Ramón Pla por el Tribunal de Responsabilidades Políticas, desligándose de la ideología de su padre y utilizando la influencia de sus contactos con personas influyentes en el nuevo régimen político. Los documentos de la “intervención de propaganda”, rellenados por los visitadores médicos en Barcelona nos permiten recuperar la opinión favorable y negativa sobre los productos del Instituto. Y gracias a los informes del agente comercial en Argentina, complementaremos el estudio del mercado barcelonés en un análisis sobre el consumo y la prescripción de los sueros Ravetllat-Pla en este país. Las estrategias de los agentes comerciales frente a las dificultades para registrar y reactivar las ventas de los medicamentos provocaron la modificación de la fórmula de los medicamentos Ravetllat-Pla. La producción de los sueros, que a principios del siglo XX se realizaba en un lugar específico desde donde se distribuían, fue transformándose hacia un nuevo modelo de producción transnacional lo que también influyó en la resignificación del medicamento al tener que adaptarse éste a las necesidades y características de los mercados según sus contextos locales. Finalmente, el fármaco fue actualizado añadiendo a la fórmula original un compuesto vitamínico dirigido al mercado nutricional. El resultado fue la elaboración de los productos «Hemo-polivit» y «Hemo-antitoxina Ravetllat-Pla (vitaminada)». Estos medicamentos fueron el resultado del proceso de reconceptualización y aparecieron a raíz de las dificultades comerciales, legales y técnicas que surgían en los diferentes mercados extranjeros y nacionales.The aim of this dissertation is to contribute to the historical comprehension of the reconceptualization of medication. This refers to the process of transformation of medicines that were initially devised and produced to treat a concrete illness and ended up used as therapies for other pathologies. Our approach involves the case study of the Ravetllat-Pla Institute between 1939 and 1955 and focuses on the production, distribution and commercialization of its two main drugs: the «Hemo-antitoxina» and the «Suero Ravetllat-Pla». The history of medicine and the history of pharmacology have studied medication from many different points of view, describing them as industrialized therapeutic products, regulated, controlled and normalized pharmaceutical developments, and/or commodities. However, the study of the reconceptualization of medication needs the merging of all these perspectives to yield a polyhedral view. From this multifaceted point of view, the interdisciplinary study of pharmacological remedies has also allowed the exploration of the evolution and transformation of the contemporary society through the complexity of all the factors involved in the reconceptualization of medication. The correspondence between sales representatives and the Institute has led to the analysis of the debate that took place around the drugs in the midst of its commercial network. The history of the Ravetllat-Pla Institute is unique due to the heterodoxy of its directors. The physician Ramón Pla Armengol (1880-1956) and the veterinarian Joaquín Ravetllat Estech (1871-1923) founded this laboratory in 1923 to produce the Hemo-antitoxina and the Suero Ravetllat-Pla. The foundational purpose of the laboratory was to legitimize the Ravetllat-Pla theory through the manufacture and sale of anti-tuberculosis products based upon this theory. In this dissertation, we examine Ramón Pla’s political career from his election as socialist representative in the 1936 parliament to his subsequent political exile. In this first part, we analyze his political and medical thinking in the texts published during his exile. This analysis allows an approach to his heterodox and critical thinking regarding the “official science” as well as his advocacy of serotherapy and clinical practice in the moment when antibiotics were gaining weight in medical debates. Political publications in Republican magazines show a disappointed Ramón Pla with the politics of Republican exile. On the other hand, while Ramón Pla was exiled, his daughter Nuria Pla Montseny (1918-2011) left to Burgos and joined the Sección Femenina of the FET de las JONS. In 1939, upon the entrance of Franco’s troops in Barcelona, Nuria Pla started directing the Ravetllat-Pla Institute, which was immersed in a deplorable scientific, economic and commercial situation. Nuria Pla’s strategy to recover the scientific and commercial activity of the Ravetllat-Pla Institute began with the defense of her ownership, which was questioned by the Court of Political Liability inquiry on Ramón Pla, by detaching herself from her father’s ideology and using his contacts’ influence with powerful characters in the new political regime. The documents corresponding to the “propaganda intervention”, filled up by its salesmen, allow the recovery of favorable and unfavorable opinions on the Institute products on the part of physicians in Barcelona as well as of patients consuming them. Also, the analysis of the consumption and prescription of Ravetllat-Pla sera extends to Argentina owing to the reports filed by the salesman in that country. Salesmen’s strategies facing obstacles to register and reactivate drugs sales led to the modification of the formulae of Ravetllat-Pla medicines. Production of sera, that was taking place in a specific location from where they were distributed at the beginning of the 20th century, changed towards a new model of transnational production which influenced the drug re-signification in order to adapt to the needs and traits of distinct local markets. Finally, adding a vitamin compound to the original formula in order to appeal to the nutritional market actualized the drug. The result was the manufacture of Ravetllat-Pla «Hemo-polivit» and «Hemo-antitoxina (vitaminada)». These drugs were the result of a process of reconceptualization and came out owing to the commercial, legal and technical difficulties arising in different national and foreign markets

El Instituto Ravetllat-Pla durante el franquismo (1939-1955): Estrategias comerciales y científicas del medicamento y la reconceptualización de la sueroterapia

El objetivo de esta memoria es contribuir a la comprensión histórica de la reconceptualización del medicamento. Nos referimos al fenómeno de transformación sufrido por aquellos fármacos que fueron pensados y producidos inicialmente para tratar una enfermedad concreta y que terminaron usándose en la terapéutica de otros procesos patológicos. Para el acercamiento que se propone, se ha utilizado el caso de estudio del Instituto Ravetllat-Pla entre 1939 y 1955 interesándonos por la producción, distribución y comercialización, fundamentalmente, de sus dos principales medicamentos: la «Hemo-antitoxina» y el «Suero Ravetllat-Pla». La historia de la medicina y de la farmacia han estudiado los medicamentos desde muy diversos ángulos y los han descrito como productos terapéuticos industrializados, como elaboraciones farmacéuticas reguladas, controladas y normalizadas y/o como artículos de consumo. Sin embargo, el estudio de la reconceptualización del medicamento necesita una integración de estas perspectivas de manera que la visión que nos ofrezca sea poliédrica. La correspondencia entre los agentes comerciales y el propio Instituto, nos han permitido analizar el debate que se produjo sobre el medicamento en el seno de su red comercial. La historia del Instituto Ravetllat-Pla resulta excepcional por la heterodoxia de sus directores. Este laboratorio se fundó en 1923 por el médico Ramón Pla Armengol (1880-1956) y el veterinario Joaquín Ravetllat Estech (1871-1923) para producir la Hemo-antitoxina y el Suero Ravetllat-Pla. El objetivo fundacional de este laboratorio fue legitimar la teoría Ravetllat-Pla a través de la fabricación y venta de los productos antituberculosos fundamentados en ella. En esta memoria estudiamos la trayectoria política de Ramón Pla hasta llegar a ser electo como diputado socialista a la Cortes en 1936 hasta su posterior exilio político. En esta primera parte analizamos su pensamiento heterodoxo y crítico con la "ciencia oficial" y también a su defensa de la seroterapia y la clínica en un momento que los antibióticos tomaban un mayor protagonismo en el debate médico. Por otro lado, mientras Ramón Pla se hallaba en el exilio, su hija Nuria Pla Monseny (1918-2011) partió a Burgos e ingresó en la Sección Femenina de la FET y de las JONS. En 1939, con la entrada de las tropas franquistas a Barcelona, Nuria Pla empezó a dirigir el Instituto Ravetllat-Pla que se encontraba en una situación científica, económica y comercial deplorable. La estrategia de Nuria Pla para lograr recuperar la actividad científica y comercial del Instituto Ravetllat-Pla comenzó por defender la propiedad del mismo, puesta en duda por el expediente abierto a Ramón Pla por el Tribunal de Responsabilidades Políticas, desligándose de la ideología de su padre y utilizando la influencia de sus contactos con personas influyentes en el nuevo régimen político. Los documentos de la "intervención de propaganda", rellenados por los visitadores médicos en Barcelona nos permiten recuperar la opinión favorable y negativa sobre los productos del Instituto. Y gracias a los informes del agente comercial en Argentina, complementaremos el estudio del mercado barcelonés en un análisis sobre el consumo y la prescripción de los sueros Ravetllat-Pla en este país. Las estrategias de los agentes comerciales frente a las dificultades para registrar y reactivar las ventas de los medicamentos provocaron la modificación de la fórmula de los medicamentos Ravetllat-Pla. La producción de los sueros, que a principios del siglo XX se realizaba en un lugar específico desde donde se distribuían, fue transformándose hacia un nuevo modelo de producción transnacional lo que también influyó en la resignificación del medicamento al tener que adaptarse éste a las necesidades y características de los mercados según sus contextos locales. Finalmente, el fármaco fue actualizado añadiendo a la fórmula original un compuesto vitamínico dirigido al mercado nutricional. El resultado fue la elaboración de los productos «Hemo-polivit» y «Hemo-antitoxina Ravetllat-Pla (vitaminada)». Estos medicamentos fueron el resultado del proceso de reconceptualización y aparecieron a raíz de las dificultades comerciales, legales y técnicas que surgían en los diferentes mercados extranjeros y nacionales.The aim of this dissertation is to contribute to the historical comprehension of the reconceptualization of medication. This refers to the process of transformation of medicines that were initially devised and produced to treat a concrete illness and ended up used as therapies for other pathologies. Our approach involves the case study of the Ravetllat-Pla Institute between 1939 and 1955 and focuses on the production, distribution and commercialization of its two main drugs: the «Hemo-antitoxina» and the «Suero Ravetllat-Pla». The history of medicine and the history of pharmacology have studied medication from many different points of view, describing them as industrialized therapeutic products, regulated, controlled and normalized pharmaceutical developments, and/or commodities. However, the study of the reconceptualization of medication needs the merging of all these perspectives to yield a polyhedral view. From this multifaceted point of view, the interdisciplinary study of pharmacological remedies has also allowed the exploration of the evolution and transformation of the contemporary society through the complexity of all the factors involved in the reconceptualization of medication. The correspondence between sales representatives and the Institute has led to the analysis of the debate that took place around the drugs in the midst of its commercial network. The history of the Ravetllat-Pla Institute is unique due to the heterodoxy of its directors. The physician Ramón Pla Armengol (1880-1956) and the veterinarian Joaquín Ravetllat Estech (1871-1923) founded this laboratory in 1923 to produce the Hemo-antitoxina and the Suero Ravetllat-Pla. The foundational purpose of the laboratory was to legitimize the Ravetllat-Pla theory through the manufacture and sale of anti-tuberculosis products based upon this theory. In this dissertation, we examine Ramón Pla's political career from his election as socialist representative in the 1936 parliament to his subsequent political exile. In this first part, we analyze his political and medical thinking in the texts published during his exile. This analysis allows an approach to his heterodox and critical thinking regarding the "official science" as well as his advocacy of serotherapy and clinical practice in the moment when antibiotics were gaining weight in medical debates. Political publications in Republican magazines show a disappointed Ramón Pla with the politics of Republican exile. On the other hand, while Ramón Pla was exiled, his daughter Nuria Pla Montseny (1918-2011) left to Burgos and joined the Sección Femenina of the FET de las JONS. In 1939, upon the entrance of Franco's troops in Barcelona, Nuria Pla started directing the Ravetllat-Pla Institute, which was immersed in a deplorable scientific, economic and commercial situation. Nuria Pla's strategy to recover the scientific and commercial activity of the Ravetllat-Pla Institute began with the defense of her ownership, which was questioned by the Court of Political Liability inquiry on Ramón Pla, by detaching herself from her father's ideology and using his contacts' influence with powerful characters in the new political regime. The documents corresponding to the "propaganda intervention", filled up by its salesmen, allow the recovery of favorable and unfavorable opinions on the Institute products on the part of physicians in Barcelona as well as of patients consuming them. Also, the analysis of the consumption and prescription of Ravetllat-Pla sera extends to Argentina owing to the reports filed by the salesman in that country. Salesmen's strategies facing obstacles to register and reactivate drugs sales led to the modification of the formulae of Ravetllat-Pla medicines. Production of sera, that was taking place in a specific location from where they were distributed at the beginning of the 20th century, changed towards a new model of transnational production which influenced the drug re-signification in order to adapt to the needs and traits of distinct local markets. Finally, adding a vitamin compound to the original formula in order to appeal to the nutritional market actualized the drug. The result was the manufacture of Ravetllat-Pla «Hemo-polivit» and «Hemo-antitoxina (vitaminada)». These drugs were the result of a process of reconceptualization and came out owing to the commercial, legal and technical difficulties arising in different national and foreign markets

MarIA: Modelos del Lenguaje en Español

This work presents MarIA, a family of Spanish language models and associated resources made available to the industry and the research community. Currently, MarIA includes RoBERTa-base, RoBERTa-large, GPT2 and GPT2-large Spanish language models, which can arguably be presented as the largest and most proficient language models in Spanish. The models were pretrained using a massive corpus of 570GB of clean and deduplicated texts with 135 billion words extracted from the Spanish Web Archive crawled by the National Library of Spain between 2009 and 2019. We assessed the performance of the models with nine existing evaluation datasets and with a novel extractive Question Answering dataset created ex novo. Overall, MarIA models outperform the existing Spanish models across a variety of NLU tasks and training settings.En este artículo se presenta MarIA, una familia de modelos del lenguaje en español y sus correspondientes recursos que se hacen públicos para la industria y la comunidad científica. Actualmente MarIA incluye los modelos del lenguaje en español RoBERTa-base, RoBERTa-large, GPT2 y GPT2-large que pueden considerarse como los modelos más grandes y mejores para español. Los modelos han sido preentrenados utilizando un corpus masivo de 570GB de textos limpios y deduplicados, que comprende un total de 135 mil millones de palabras extraidas del Archivo Web del Español construido por la Biblioteca Nacional de España entre los años 2009 y 2019. Evaluamos el rendimiento de los modelos con nueve conjuntos de datos existentes y con un nuevo conjunto de datos de pregunta-respuesta extractivo creado ex novo. El conjunto de modelos de MarIA supera, en la practica totalidad, el rendimiento de los modelos existentes en español en las diferentes tareas y configuraciones presentadas.This work was funded by the Spanish State Secretariat for Digitalization and Artificial Intelligence (SEDIA) within the framework of the Plan-TL

Enhanced MAF Oncogene Expression and Breast Cancer Bone Metastasis.

BACKGROUND: There are currently no biomarkers for early breast cancer patient populations at risk of bone metastasis. Identification of mediators of bone metastasis could be of clinical interest. METHODS: A de novo unbiased screening approach based on selection of highly bone metastatic breast cancer cells in vivo was used to determine copy number aberrations (CNAs) associated with bone metastasis. The CNAs associated with bone metastasis were examined in independent primary breast cancer datasets with annotated clinical follow-up. The MAF gene encoded within the CNA associated with bone metastasis was subjected to gain and loss of function validation in breast cancer cells (MCF7, T47D, ZR-75, and 4T1), its downstream mechanism validated, and tested in clinical samples. A multivariable Cox cause-specific hazard model with competing events (death) was used to test the association between 16q23 or MAF and bone metastasis. All statistical tests were two-sided. RESULTS: 16q23 gain CNA encoding the transcription factor MAF mediates breast cancer bone metastasis through the control of PTHrP. 16q23 gain (hazard ratio (HR) for bone metastasis = 14.5, 95% confidence interval (CI) = 6.4 to 32.9, P < .001) as well as MAF overexpression (HR for bone metastasis = 2.5, 95% CI = 1.7 to 3.8, P < .001) in primary breast tumors were specifically associated with risk of metastasis to bone but not to other organs. CONCLUSIONS: These results suggest that MAF is a mediator of breast cancer bone metastasis. 16q23 gain or MAF protein overexpression in tumors may help to select patients at risk of bone relapse.MP and SG is supported by “La Caixa” PhD fellowship program. AAE and AB hold PhD fellowships from the Spanish Ministerio de Ciencia e Innovación (MICINN). MTS is supported by the IRB Barcelona PhD program. JU is a “Juan de la Cierva” Researcher (MICINN). FIS PI12/00680, PI12/01552, and PI12/01421 supported JA, ALl, and FR, respectively. JA, ALl, and AP were part of RD12/0036/0051, RD12/0036/0070, and RD12/0036/0042, and FR is part of biobanc RD/09/0076/00101. JA and FR are recipients of intensification grant ISCIII, 2009SGR321, XBTC, MARBiobanc, and Cellex. RRG was supported by the Institució Catalana de Recerca i Estudis Avançats. Support and structural funds were provided by the BBVA Foundation, the Generalitat de Catalunya (2014 SGR 535), and the Spanish Ministerio de Ciencia e Innovación (MICINN) (SAF2013-46196) to RRG

Enhanced MAF Oncogene Expression and Breast Cancer Bone Metastasis

Background: There are currently no biomarkers for early breast cancer patient populations at risk of bone metastasis. Identification of mediators of bone metastasis could be of clinical interest. Methods: A de novo unbiased screening approach based on selection of highly bone metastatic breast cancer cells in vivo was used to determine copy number aberrations (CNAs) associated with bone metastasis. The CNAs associated with bone metastasis were examined in independent primary breast cancer datasets with annotated clinical follow-up. The MAF gene encoded within the CNA associated with bone metastasis was subjected to gain and loss of function validation in breast cancer cells (MCF7, T47D, ZR-75, and 4T1), its downstream mechanism validated, and tested in clinical samples. A multivariable Cox cause-specific hazard model with competing events (death) was used to test the association between 16q23 or MAF and bone metastasis. All statistical tests were two-sided. Results: 16q23 gain CNA encoding the transcription factor MAF mediates breast cancer bone metastasis through the control of PTHrP. 16q23 gain (hazard ratio (HR) for bone metastasis = 14.5, 95% confidence interval (CI) = 6.4 to 32.9, P < .001) as well as MAF overexpression (HR for bone metastasis = 2.5, 95% CI = 1.7 to 3.8, P < .001) in primary breast tumors were specifically associated with risk of metastasis to bone but not to other organs. Conclusions: These results suggest that MAF is a mediator of breast cancer bone metastasis. 16q23 gain or MAF protein overexpression in tumors may help to select patients at risk of bone relapse

Enhanced MAF Oncogene Expression and Breast Cancer Bone Metastasis.

BACKGROUND: There are currently no biomarkers for early breast cancer patient populations at risk of bone metastasis. Identification of mediators of bone metastasis could be of clinical interest. METHODS: A de novo unbiased screening approach based on selection of highly bone metastatic breast cancer cells in vivo was used to determine copy number aberrations (CNAs) associated with bone metastasis. The CNAs associated with bone metastasis were examined in independent primary breast cancer datasets with annotated clinical follow-up. The MAF gene encoded within the CNA associated with bone metastasis was subjected to gain and loss of function validation in breast cancer cells (MCF7, T47D, ZR-75, and 4T1), its downstream mechanism validated, and tested in clinical samples. A multivariable Cox cause-specific hazard model with competing events (death) was used to test the association between 16q23 or MAF and bone metastasis. All statistical tests were two-sided. RESULTS: 16q23 gain CNA encoding the transcription factor MAF mediates breast cancer bone metastasis through the control of PTHrP. 16q23 gain (hazard ratio (HR) for bone metastasis = 14.5, 95% confidence interval (CI) = 6.4 to 32.9, P < .001) as well as MAF overexpression (HR for bone metastasis = 2.5, 95% CI = 1.7 to 3.8, P < .001) in primary breast tumors were specifically associated with risk of metastasis to bone but not to other organs. CONCLUSIONS: These results suggest that MAF is a mediator of breast cancer bone metastasis. 16q23 gain or MAF protein overexpression in tumors may help to select patients at risk of bone relapse.MP and SG is supported by “La Caixa” PhD fellowship program. AAE and AB hold PhD fellowships from the Spanish Ministerio de Ciencia e Innovación (MICINN). MTS is supported by the IRB Barcelona PhD program. JU is a “Juan de la Cierva” Researcher (MICINN). FIS PI12/00680, PI12/01552, and PI12/01421 supported JA, ALl, and FR, respectively. JA, ALl, and AP were part of RD12/0036/0051, RD12/0036/0070, and RD12/0036/0042, and FR is part of biobanc RD/09/0076/00101. JA and FR are recipients of intensification grant ISCIII, 2009SGR321, XBTC, MARBiobanc, and Cellex. RRG was supported by the Institució Catalana de Recerca i Estudis Avançats. Support and structural funds were provided by the BBVA Foundation, the Generalitat de Catalunya (2014 SGR 535), and the Spanish Ministerio de Ciencia e Innovación (MICINN) (SAF2013-46196) to RRG