Can language models automate data wrangling?

[ES] La automatización de la ciencia de datos y otros procesos de manipulación de datos dependen de la integración y el formateo de los datos "desordenados". La manipulación de datos es un término que engloba estas tareas tediosas y que requieren mucho tiempo. Tareas como la transformación de fechas, unidades o nombres expresados en diferentes formatos han sido un reto para el aprendizaje automático porque los usuarios esperan resolverlas con pistas cortas o pocos ejemplos, y los problemas dependen en gran medida del conocimiento del dominio. Curiosamente, los grandes modelos lingüísticos de hoy en día infieren a partir de muy pocos ejemplos o incluso de una breve pista en lenguaje natural, e integran grandes cantidades de conocimiento del dominio. Por tanto, es una cuestión de investigación importante analizar si los modelos de lenguaje son un enfoque prometedor para la gestión de datos, especialmente porque sus capacidades siguen creciendo. En este artículo aplicamos diferentes variantes de modelos lingüísticos de GPT a problemas de gestión de datos, comparando sus resultados con los de herramientas especializadas de gestión de datos, y analizando también las tendencias, variaciones y nuevas posibilidades y riesgos de los modelos lingüísticos en esta tarea. Nuestro principal hallazgo es que parecen ser una herramienta poderosa para una amplia gama de tareas de búsqueda de datos, pero la fiabilidad puede ser un problema importante a superar.[EN] The automation of data science and other data manipulation processes depend on the integration and formatting of ‘messy’ data. Data wran gling is an umbrella term for these tedious and time-consuming tasks. Tasks such as transforming dates, units or names expressed in different formats have been challenging for machine learning because users expect to solve them with short cues or few examples, and the problems depend heavily on domain knowledge. Interestingly, large language models today infer from very few examples or even a short clue in natural language, and integrate vast amounts of domain knowledge. It is then an important research question to analyse whether language models are a promising approach for data wrangling, especially as their capabilities continue growing. In this paper we apply different language model variants of GPT to data wrangling problems, comparing their results to specialised data wrangling tools, also analysing the trends, variations and further possibilities and risks of language models in this task. Our major finding is that they appear as a powerful tool for a wide range of data wrangling tasks, but reliability may be an important issue to overcome.Jaimovitch-López, G.; Ferri, C.; Hernández-Orallo, J.; Martínez-Plumed, F.; Ramírez-Quintana, MJ. (2021). Can language models automate data wrangling?. http://hdl.handle.net/10251/18502

Can language models automate data wrangling?

[EN] The automation of data science and other data manipulation processes depend on the integration and formatting of 'messy' data. Data wrangling is an umbrella term for these tedious and time-consuming tasks. Tasks such as transforming dates, units or names expressed in different formats have been challenging for machine learning because (1) users expect to solve them with short cues or few examples, and (2) the problems depend heavily on domain knowledge. Interestingly, large language models today (1) can infer from very few examples or even a short clue in natural language, and (2) can integrate vast amounts of domain knowledge. It is then an important research question to analyse whether language models are a promising approach for data wrangling, especially as their capabilities continue growing. In this paper we apply different variants of the language model Generative Pre-trained Transformer (GPT) to five batteries covering a wide range of data wrangling problems. We compare the effect of prompts and few-shot regimes on their results and how they compare with specialised data wrangling systems and other tools. Our major finding is that they appear as a powerful tool for a wide range of data wrangling tasks. We provide some guidelines about how they can be integrated into data processing pipelines, provided the users can take advantage of their flexibility and the diversity of tasks to be addressed. However, reliability is still an important issue to overcome.Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. This work was funded by the Future of Life Institute, FLI, under grant RFP2-152, the MIT-Spain - INDITEX Sustainability Seed Fund under project COST-OMIZE, the EU (FEDER) and Spanish MINECO under RTI2018-094403-B-C32 and PID2021-122830OB-C42, Generalitat Valenciana under PROMETEO/2019/098 and INNEST/2021/317, EU's Horizon 2020 research and innovation programme under grant agreement No. 952215 (TAILOR) and US DARPA HR00112120007 ReCOG-AI. AcknowledgementsWe thank Lidia Contreras for her help with the Data Wrangling Dataset Repository. We thank the anonymous reviewers from ECMLPKDD Workshop on Automating Data Science (ADS2021) and the anonymous reviewers of this special issue for their comments.Jaimovitch-López, G.; Ferri Ramírez, C.; Hernández-Orallo, J.; Martínez-Plumed, F.; Ramírez Quintana, MJ. (2023). Can language models automate data wrangling?. Machine Learning. 112(6):2053-2082. https://doi.org/10.1007/s10994-022-06259-920532082112

Adaptación del modelo de planificación de la producción al contexto de la red de distribución en una planta química

[ES] La digitalización ha supuesto un aumento exponencial en las capacidades tanto productivas como de toma de decisiones en las organizaciones. Los sistemas de información proveen a responsables en todos los niveles de la jerarquía empresarial de información relevante y actualizada. En este trabajo se expone un caso real en una empresa que opera en el sector conocido como "Automotive Coatings". El objetivo principal es ampliar el modelo actualmente implantado de planificación de la producción en la planta de Valencia, aprovechando la información disponible sobre los niveles de inventario en los diferentes nodos que conforman la red de distribución con el fin de conseguir una planificación más acertada.[EN] Digitalization has led to an exponential increase in both productive and decision-making capacities in organizations. Information systems provide managers at all levels of the business hierarchy with relevant and up-to-date information. This paper presents a real case in a company operating in the sector known as "Automotive Coatings". The main objective is to extend the currently implemented production planning model in the Valencia plant, taking advantage of the information available on inventory levels in the different nodes that comprise the distribution network in order to achieve more accurate planning.Jaimovitch López, GE. (2020). Adaptación del modelo de planificación de la producción al contexto de la red de distribución en una planta química. http://hdl.handle.net/10251/152559TFG

Comparación entre el aprendizaje de machine learning y humanos desde ejemplos generados con machine teaching

[EN] In contrast to the term widely known as Machine Learning, a successful branch of Artificial Intelligence (AI), the concept of Machine Teaching arises. One of its main objectives is the optimization of learning through the choice of labeled examples that will constitute the training set for the learning models. Recently, some approaches have focused on the use of these techniques to obtain examples for human learning. This application is strongly related to the field of explainable AI, more specifically to exemplar-based explanations, whose purpose is to convey to humans what a machine has learned. In this paper we propose to make a comparison of the learning process from examples generated by Machine Teaching among different learning systems like an inductive functional programming system (MagicHaskeller), a transformer-based deep neural network (GPT-2) and humans. To conclude, the effectiveness of the exemplar-based explanations using this setting is discussed. The obtained results highlight the necessity of providing additional information alongside the optimal example sets, extracted using the machine teaching setting applied in this work.[ES] En contraste al término ampliamente conocido como aprendizaje automático (Machine Learning), rama de éxito en el campo de la Inteligencia Artificial (IA), surge el concepto de enseñanza automática (Machine Teaching). Uno de sus objetivos principales es la optimización del aprendizaje mediante la elección de ejemplos etiquetados que constituirán el conjunto de entrenamiento para los modelos de aprendizaje. Recientemente, se han empezado a desarrollar aproximaciones enfocadas a la utilización de estas técnicas de obtención de ejemplos centradas en el aprendizaje de humanos. Esta aplicación está fuertemente relacionada con el concepto de IA explicable, más concretamente con las explicaciones basadas en ejemplos, cuya finalidad es transmitir a los humanos aquello que una máquina ha aprendido. En este trabajo se propone realizar una comparación del proceso de aprendizaje desde ejemplos generados por Machine Teaching entre diferentes sistemas de aprendizaje como un sistema de programación funcional inductiva (MagicHaskeller), una red neuronal profunda basada en el modelo Transformer (GPT-2) y humanos. Para concluir, la efectividad de las explicaciones basadas en ejemplos con esta configuración es analizada. Los resultados obtenidos señalan la necesidad de proporcionar información adicional junto a los conjuntos óptimos de ejemplos, extraídos mediante la configuración de Machine Teaching aplicada en este trabajo.[CA] En contrast al terme àmpliament conegut com aprenentatge automàtic (Machine Learning), branca d’èxit en el camp de la Intel·ligència Artificial (IA), sorgeix el concepte d’ensenyament automàtic (Machine Teaching). Un dels seus objectius principals és l’optimització de l’aprenentatge mitjançant l’elecció d’exemples etiquetats que constituiran el conjunt d’entrenament per als models d’aprenentatge. Recentment, s’han començat a desenvolupar aproximacions enfocades a la utilització d’aquestes tècniques d’obtenció d’exemples centrades en l’aprenentatge d’humans. Aquesta aplicació està fortament relacionada amb la branca de IA explicable, més concretament amb les explicacions basades en exemples, la finalitat dels quals és transmetre als humans allò que una màquina ha aprés. En aquest treball es proposa realitzar una comparació del procés d’aprenentatge des d’exemples generats per Machine Teaching entre diferents sistemes d’aprenentatge com un sistema de programació funcional inductiva (MagicHaskeller), una xarxa neuronal profunda basada en el model Transformer (GPT-2) i humans. Per a concloure, l’efectivitat de les explicacions basades en exemples amb aquesta configuració és analitzada. Els resultats obtinguts assenyalen la necessitat de proporcionar informació addicional junt als conjunts òptims d’exemples, extrets mitjançant la configuració de Machine Teaching aplicada en aquest treball.The author was partially funded by the DMIP group of the Valencian Research Institute for Artificial Intelligence (VRAIN)Jaimovitch López, GE. (2020). Comparison between machine learning and human learning from examples generated with machine teaching. http://hdl.handle.net/10251/152771TFG

Beyond the imitation game: Quantifying and extrapolating the capabilities of language models

Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG-bench). BIG-bench currently consists of 204 tasks, contributed by 442 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit "breakthrough" behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting

Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG-bench). BIG-bench currently consists of 204 tasks, contributed by 442 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit "breakthrough" behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting.Comment: 27 pages, 17 figures + references and appendices, repo: https://github.com/google/BIG-benc