Cappy: Outperforming and Boosting Large Multi-Task LMs with a Small Scorer

Large language models (LLMs) such as T0, FLAN, and OPT-IML, excel in multi-tasking under a unified instruction-following paradigm, where they also exhibit remarkable generalization abilities to unseen tasks. Despite their impressive performance, these LLMs, with sizes ranging from several billion to hundreds of billions of parameters, demand substantial computational resources, making their training and inference expensive and inefficient. Furthermore, adapting these models to downstream applications, particularly complex tasks, is often unfeasible due to the extensive hardware requirements for finetuning, even when utilizing parameter-efficient approaches such as prompt tuning. Additionally, the most powerful multi-task LLMs, such as OPT-IML-175B and FLAN-PaLM-540B, are not publicly accessible, severely limiting their customization potential. To address these challenges, we introduce a pretrained small scorer, Cappy, designed to enhance the performance and efficiency of multi-task LLMs. With merely 360 million parameters, Cappy functions either independently on classification tasks or serve as an auxiliary component for LLMs, boosting their performance. Moreover, Cappy enables efficiently integrating downstream supervision without requiring LLM finetuning nor the access to their parameters. Our experiments demonstrate that, when working independently on 11 language understanding tasks from PromptSource, Cappy outperforms LLMs that are several orders of magnitude larger. Besides, on 45 complex tasks from BIG-Bench, Cappy boosts the performance of the advanced multi-task LLM, FLAN-T5, by a large margin. Furthermore, Cappy is flexible to cooperate with other LLM adaptations, including finetuning and in-context learning, offering additional performance enhancement.Comment: In proceedings of NeurIPS 2023; Code and model available at https://github.com/tanyuqian/cappy and https://huggingface.co/btan2/cappy-large, respectivel

Redco: A Lightweight Tool to Automate Distributed Training of LLMs on Any GPU/TPUs

The recent progress of AI can be largely attributed to large language models (LLMs). However, their escalating memory requirements introduce challenges for machine learning (ML) researchers and engineers. Addressing this requires developers to partition a large model to distribute it across multiple GPUs or TPUs. This necessitates considerable coding and intricate configuration efforts with existing model parallel tools, such as Megatron-LM, DeepSpeed, and Alpa. These tools require users' expertise in machine learning systems (MLSys), creating a bottleneck in LLM development, particularly for developers without MLSys background. In this work, we present Redco, a lightweight and user-friendly tool crafted to automate distributed training and inference for LLMs, as well as to simplify ML pipeline development. The design of Redco emphasizes two key aspects. Firstly, to automate model parallism, our study identifies two straightforward rules to generate tensor parallel strategies for any given LLM. Integrating these rules into Redco facilitates effortless distributed LLM training and inference, eliminating the need of additional coding or complex configurations. We demonstrate the effectiveness by applying Redco on a set of LLM architectures, such as GPT-J, LLaMA, T5, and OPT, up to the size of 66B. Secondly, we propose a mechanism that allows for the customization of diverse ML pipelines through the definition of merely three functions, eliminating redundant and formulaic code like multi-host related processing. This mechanism proves adaptable across a spectrum of ML algorithms, from foundational language modeling to complex algorithms like meta-learning and reinforcement learning. Consequently, Redco implementations exhibit much fewer code lines compared to their official counterparts.Comment: Released under Apache License 2.0 at https://github.com/tanyuqian/redc

A solvent evaporation route towards fabrication of hierarchically porous ZSM-11 with highly accessible mesopores

A solvent evaporation route to generate an organosilane modified dry gel and its transformation into hierarchically porous ZSM-11 is reported. The material features good pore-connectivity and improved acid site accessibility towards bulky substrates.</p

Portability and networked learning environments

Abstract The portability of educational software is defined as the likelihood of software usage, with or without adaptation, in an educational environment different from that for which it was originally designed and produced. Barriers and research relevant to the portability of electronic learning resources are discussed and organised into a portability-limiting factors model. With the increase in number and scope of networked learning environments, portability issues take on a new dimension. Using electronic (study) books as an example, the portability problem space of networked learning environments is explored

Phase Stability of Hexagonal/cubic Boron Nitride Nanocomposites

Boron nitride (BN) is an exceptional material and among its polymorphs, two-dimensional (2D) hexagonal and three-dimensional (3D) cubic BN (h-BN and c-BN) phases are most common. The phase stability regimes of these BN phases are still under debate and phase transformations of h-BN/c-BN remain a topic of interest. Here, we investigate the phase stability of 2D/3D h-BN/c-BN nanocomposites and show that the co-existence of two phases can lead to strong non-linear optical properties and low thermal conductivity at room temperature. Furthermore, spark-plasma sintering of the nanocomposite shows complete phase transformation to 2D h-BN with improved crystalline quality, where 3D c-BN grain sizes governs the nucleation and growth kinetics. Our demonstration might be insightful in phase engineering of BN polymorphs based nanocomposites with desirable properties for optoelectronics and thermal energy management applications.Comment: 29 pages, 5 figure