Self-Supervised Learning with an Information Maximization Criterion

Self-supervised learning allows AI systems to learn effective representations from large amounts of data using tasks that do not require costly labeling. Mode collapse, i.e., the model producing identical representations for all inputs, is a central problem to many self-supervised learning approaches, making self-supervised tasks, such as matching distorted variants of the inputs, ineffective. In this article, we argue that a straightforward application of information maximization among alternative latent representations of the same input naturally solves the collapse problem and achieves competitive empirical results. We propose a self-supervised learning method, CorInfoMax, that uses a second-order statistics-based mutual information measure that reflects the level of correlation among its arguments. Maximizing this correlative information measure between alternative representations of the same input serves two purposes: (1) it avoids the collapse problem by generating feature vectors with non-degenerate covariances; (2) it establishes relevance among alternative representations by increasing the linear dependence among them. An approximation of the proposed information maximization objective simplifies to a Euclidean distance-based objective function regularized by the log-determinant of the feature covariance matrix. The regularization term acts as a natural barrier against feature space degeneracy. Consequently, beyond avoiding complete output collapse to a single point, the proposed approach also prevents dimensional collapse by encouraging the spread of information across the whole feature space. Numerical experiments demonstrate that CorInfoMax achieves better or competitive performance results relative to the state-of-the-art SSL approaches

Performance analysis of vertical handover using predictable LGD event based on IEEE 802.21.

Next Generation Wireless Networks (NGWN) aim to provide any service at any time and anywhere with seamless mobility between homogeneous and heterogeneous networks. IEEE defines the IEEE 802.21 standard to facilitate seamless handover, namely, Media Independent Handover (MIH). IEEE 802.21 provides layer two events to upper layers with a view to enhance the operability and enable them to make the right decision on time. Link Going Down (LGD) is a predictive event triggered when a link quality degradation is expected in the near future. Connectivity losses and quality decreases are usually foreseeable during the handover process. Therefore, in this paper, we analyze the performance of our effective prediction model for generating the Link Going Down (LGD) event. The network performance metrics, such as packet loss, end-to-end delay, and throughput, have been evaluated using the Network Simulator NS2

Unraveling flux behavior of superhydrophilic loose nanofiltration membranes during textile wastewater treatment

© 2015 Elsevier B.V. Loose nanofiltration (NF) membranes can be used to treat textile wastewater effectively, providing an attracted avenue for resource recovery (i.e., dye purification and salt reuse) at the premise of the integration of high dye retention and salt permeation. However, the issue of membrane fouling has to be adequately addressed in view of its practical application. In this study, superhydrophilic loose NF membranes (Sepro NF 6 and NF 2A, Ultura) were applied for textile wastewater treatment. Synthetic solutions containing dyes and NaCl were used as the feed stream of a NF unit. It was found that two factors, namely cake-enhanced concentration polarization and the formation of a dye cake layer, dramatically deteriorated the flux of NF membranes with a synergic effect. In viewpoint of realistic application, diafiltration of a binary dye/salt mixture indicates that cake-enhanced concentration polarization plays a dominant role for the low membrane flux. As the diafiltration continued, cake-enhanced concentration polarization was alleviated with a decreasing concentration of salt in the feed. At the subsequent post-concentration procedure, the formation of a dye cake layer slightly compromised the membrane flux, but the negative impact of cake-enhanced concentration polarization was negligible due to the small quantity of salt remained in the feed.status: publishe

Beyond the imitation game: Quantifying and extrapolating the capabilities of language models

Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG-bench). BIG-bench currently consists of 204 tasks, contributed by 442 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit "breakthrough" behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting

Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG-bench). BIG-bench currently consists of 204 tasks, contributed by 442 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit "breakthrough" behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting.Comment: 27 pages, 17 figures + references and appendices, repo: https://github.com/google/BIG-benc