Self-Supervised Learning with an Information Maximization Criterion

Self-supervised learning allows AI systems to learn effective representations from large amounts of data using tasks that do not require costly labeling. Mode collapse, i.e., the model producing identical representations for all inputs, is a central problem to many self-supervised learning approaches, making self-supervised tasks, such as matching distorted variants of the inputs, ineffective. In this article, we argue that a straightforward application of information maximization among alternative latent representations of the same input naturally solves the collapse problem and achieves competitive empirical results. We propose a self-supervised learning method, CorInfoMax, that uses a second-order statistics-based mutual information measure that reflects the level of correlation among its arguments. Maximizing this correlative information measure between alternative representations of the same input serves two purposes: (1) it avoids the collapse problem by generating feature vectors with non-degenerate covariances; (2) it establishes relevance among alternative representations by increasing the linear dependence among them. An approximation of the proposed information maximization objective simplifies to a Euclidean distance-based objective function regularized by the log-determinant of the feature covariance matrix. The regularization term acts as a natural barrier against feature space degeneracy. Consequently, beyond avoiding complete output collapse to a single point, the proposed approach also prevents dimensional collapse by encouraging the spread of information across the whole feature space. Numerical experiments demonstrate that CorInfoMax achieves better or competitive performance results relative to the state-of-the-art SSL approaches

Performance analysis of vertical handover using predictable LGD event based on IEEE 802.21.

Next Generation Wireless Networks (NGWN) aim to provide any service at any time and anywhere with seamless mobility between homogeneous and heterogeneous networks. IEEE defines the IEEE 802.21 standard to facilitate seamless handover, namely, Media Independent Handover (MIH). IEEE 802.21 provides layer two events to upper layers with a view to enhance the operability and enable them to make the right decision on time. Link Going Down (LGD) is a predictive event triggered when a link quality degradation is expected in the near future. Connectivity losses and quality decreases are usually foreseeable during the handover process. Therefore, in this paper, we analyze the performance of our effective prediction model for generating the Link Going Down (LGD) event. The network performance metrics, such as packet loss, end-to-end delay, and throughput, have been evaluated using the Network Simulator NS2

Unraveling flux behavior of superhydrophilic loose nanofiltration membranes during textile wastewater treatment

© 2015 Elsevier B.V. Loose nanofiltration (NF) membranes can be used to treat textile wastewater effectively, providing an attracted avenue for resource recovery (i.e., dye purification and salt reuse) at the premise of the integration of high dye retention and salt permeation. However, the issue of membrane fouling has to be adequately addressed in view of its practical application. In this study, superhydrophilic loose NF membranes (Sepro NF 6 and NF 2A, Ultura) were applied for textile wastewater treatment. Synthetic solutions containing dyes and NaCl were used as the feed stream of a NF unit. It was found that two factors, namely cake-enhanced concentration polarization and the formation of a dye cake layer, dramatically deteriorated the flux of NF membranes with a synergic effect. In viewpoint of realistic application, diafiltration of a binary dye/salt mixture indicates that cake-enhanced concentration polarization plays a dominant role for the low membrane flux. As the diafiltration continued, cake-enhanced concentration polarization was alleviated with a decreasing concentration of salt in the feed. At the subsequent post-concentration procedure, the formation of a dye cake layer slightly compromised the membrane flux, but the negative impact of cake-enhanced concentration polarization was negligible due to the small quantity of salt remained in the feed.status: publishe

Beyond the imitation game: Quantifying and extrapolating the capabilities of language models

Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG-bench). BIG-bench currently consists of 204 tasks, contributed by 442 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit "breakthrough" behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting

Effects of hospital facilities on patient outcomes after cancer surgery: an international, prospective, observational study

© 2022 The Author(s). Published by Elsevier Ltd. This is an Open Access article under the CC BY 4.0 licenseBackground: Early death after cancer surgery is higher in low-income and middle-income countries (LMICs) compared with in high-income countries, yet the impact of facility characteristics on early postoperative outcomes is unknown. The aim of this study was to examine the association between hospital infrastructure, resource availability, and processes on early outcomes after cancer surgery worldwide. Methods: A multimethods analysis was performed as part of the GlobalSurg 3 study—a multicentre, international, prospective cohort study of patients who had surgery for breast, colorectal, or gastric cancer. The primary outcomes were 30-day mortality and 30-day major complication rates. Potentially beneficial hospital facilities were identified by variable selection to select those associated with 30-day mortality. Adjusted outcomes were determined using generalised estimating equations to account for patient characteristics and country-income group, with population stratification by hospital. Findings: Between April 1, 2018, and April 23, 2019, facility-level data were collected for 9685 patients across 238 hospitals in 66 countries (91 hospitals in 20 high-income countries; 57 hospitals in 19 upper-middle-income countries; and 90 hospitals in 27 low-income to lower-middle-income countries). The availability of five hospital facilities was inversely associated with mortality: ultrasound, CT scanner, critical care unit, opioid analgesia, and oncologist. After adjustment for case-mix and country income group, hospitals with three or fewer of these facilities (62 hospitals, 1294 patients) had higher mortality compared with those with four or five (adjusted odds ratio [OR] 3·85 [95% CI 2·58–5·75]; p<0·0001), with excess mortality predominantly explained by a limited capacity to rescue following the development of major complications (63·0% vs 82·7%; OR 0·35 [0·23–0·53]; p<0·0001). Across LMICs, improvements in hospital facilities would prevent one to three deaths for every 100 patients undergoing surgery for cancer. Interpretation: Hospitals with higher levels of infrastructure and resources have better outcomes after cancer surgery, independent of country income. Without urgent strengthening of hospital infrastructure and resources, the reductions in cancer-associated mortality associated with improved access will not be realised. Funding: National Institute for Health and Care Research