The Optimal Noise in Noise-Contrastive Learning Is Not What You Think

Publisher Copyright: © 2022 Proceedings of the 38th Conference on Uncertainty in Artificial Intelligence, UAI 2022. All right reserved.Learning a parametric model of a data distribution is a well-known statistical problem that has seen renewed interest as it is brought to scale in deep learning. Framing the problem as a self-supervised task, where data samples are discriminated from noise samples, is at the core of state-of-the-art methods, beginning with Noise-Contrastive Estimation (NCE). Yet, such contrastive learning requires a good noise distribution, which is hard to specify; domain-specific heuristics are therefore widely used. While a comprehensive theory is missing, it is widely assumed that the optimal noise should in practice be made equal to the data, both in distribution and proportion; this setting underlies Generative Adversarial Networks (GANs) in particular. Here, we empirically and theoretically challenge this assumption on the optimal noise. We show that deviating from this assumption can actually lead to better statistical estimators, in terms of asymptotic variance. In particular, the optimal noise distribution is different from the data's and even from a different family.Peer reviewe

Uncovering the structure of clinical EEG signals with self-supervised learning

Objective. Supervised learning paradigms are often limited by the amount of labeled data that is available. This phenomenon is particularly problematic in clinically-relevant data, such as electroencephalography (EEG), where labeling can be costly in terms of specialized expertise and human processing time. Consequently, deep learning architectures designed to learn on EEG data have yielded relatively shallow models and performances at best similar to those of traditional feature-based approaches. However, in most situations, unlabeled data is available in abundance. By extracting information from this unlabeled data, it might be possible to reach competitive performance with deep neural networks despite limited access to labels. Approach. We investigated self-supervised learning (SSL), a promising technique for discovering structure in unlabeled data, to learn representations of EEG signals. Specifically, we explored two tasks based on temporal context prediction as well as contrastive predictive coding on two clinically-relevant problems: EEG-based sleep staging and pathology detection. We conducted experiments on two large public datasets with thousands of recordings and performed baseline comparisons with purely supervised and hand-engineered approaches. Main results. Linear classifiers trained on SSL-learned features consistently outperformed purely supervised deep neural networks in low-labeled data regimes while reaching competitive performance when all labels were available. Additionally, the embeddings learned with each method revealed clear latent structures related to physiological and clinical phenomena, such as age effects. Significance. We demonstrate the benefit of SSL approaches on EEG data. Our results suggest that self-supervision may pave the way to a wider use of deep learning models on EEG data.Peer reviewe

Deep Recurrent Encoder: A scalable end-to-end network to model brain signals

Understanding how the brain responds to sensory inputs is challenging: brain recordings are partial, noisy, and high dimensional; they vary across sessions and subjects and they capture highly nonlinear dynamics. These challenges have led the community to develop a variety of preprocessing and analytical (almost exclusively linear) methods, each designed to tackle one of these issues. Instead, we propose to address these challenges through a specific end-to-end deep learning architecture, trained to predict the brain responses of multiple subjects at once. We successfully test this approach on a large cohort of magnetoencephalography (MEG) recordings acquired during a one-hour reading task. Our Deep Recurrent Encoding (DRE) architecture reliably predicts MEG responses to words with a three-fold improvement over classic linear methods. To overcome the notorious issue of interpretability of deep learning, we describe a simple variable importance analysis. When applied to DRE, this method recovers the expected evoked responses to word length and word frequency. The quantitative improvement of the present deep learning approach paves the way to better understand the nonlinear dynamics of brain activity from large datasets

The Optimal Noise in Noise-Contrastive Learning Is Not What You Think

International audienceLearning a parametric model of a data distribution is a well-known statistical problem that has seen renewed interest as it is brought to scale in deep learning. Framing the problem as a self-supervised task, where data samples are discriminated from noise samples, is at the core of state-of-the-art methods, beginning with Noise-Contrastive Estimation (NCE). Yet, such contrastive learning requires a good noise distribution, which is hard to specify; domain-specific heuristics are therefore widely used. While a comprehensive theory is missing, it is widely assumed that the optimal noise should in practice be made equal to the data, both in distribution and proportion; this setting underlies Generative Adversarial Networks (GANs) in particular. Here, we empirically and theoretically challenge this assumption on the optimal noise. We show that deviating from this assumption can actually lead to better statistical estimators, in terms of asymptotic variance. In particular, the optimal noise distribution is different from the data's and even from a different family

Incidence and clinical outcomes of nosocomial infections in patients presenting with STEMI complicated by cardiogenic shock in the United States

OBJECTIVES: This study addresses the incidence, trends, and impact of nosocomial infections (NI) on the outcomes of patients admitted with ST-segment elevation myocardial infarction (STEMI) and cardiogenic shock (STEMI-CS) using the United States National Inpatient Sample (NIS) database. METHODS: We analyzed data from 105,184 STEMI-CS patients using the NIS database from the years 2005-2014. NI was defined as infections of more than or equal to three days, comprising of central line-associated bloodstream infection (CLABSI), urinary tract infection (UTI), hospital-acquired pneumonia (HAP), Clostridium difficile infection (CDI), bacteremia, and skin related infections. Outcomes of the impact of NI on STEMI-CS included in-hospital mortality, length of hospital stay (LOS) and costs. Significant associations of NI in patients admitted with STEMI-CS were also identified. RESULTS: Overall, 19.1% (20,137) of patients admitted with STEMI-CS developed NI. Trends of NI have decreased from 2005-2014. The most common NI were UTI (9.2%), followed by HAP (6.8%), CLABSI (1.5%), bacteremia (1.5%), skin related infections (1.5%), and CDI (1.3%). The strongest association of developing a NI was increasing LOS (7-9 days; OR: 1.99; 95% CI: 1.75-2.26; \u3e9 days; OR: 4.51; 95% CI: 4.04-5.04 compared to 4-6 days as reference). Increased mortality risk among patients with NI was significant, especially those with sepsis-associated NI compared to those without sepsis (OR: 2.95; 95% CI: 2.72-3.20). Patients with NI were found to be associated with significantly longer LOS and higher costs, irrespective of percutaneous mechanical circulatory support placement. CONCLUSIONS: NI were common among patients with STEMI-CS. Those who developed NI were at a greater risk of in-hospital mortality, increased LOS and costs

Impacts of Job Standardisation on Restaurant Frontline Employees: Mediating Effect of Emotional Labour

Managers of food service operations standardise various aspects of operations to sustain consistent service quality. Frontline employees in these operations are expected to carry out tasks as per standards. Standards demand that frontline employees regulate their behaviours and emotions to complete their duties. Therefore, referring to the organisational role theory and the emotion regulation theory as the directing basis, this study examined the impact of job standardisation on emotional labour, as well as the effect of emotional labour on emotional exhaustion and job satisfaction of frontline employees in the hospitality sector. This study also examined the mediating effect of emotional labour on the relation between job standardisation, on one hand, and emotional exhaustion and job satisfaction on the other hand. The data collection was carried out in food service operations in Lebanon. Structural equation modelling (SEM) was used to assess the relations. The results showed that job standardisation negatively affected emotional labour and that emotional labour had a positive effect on emotional exhaustion and a negative effect on job satisfaction. Furthermore, emotional labour mediated the relation between job standardisation and emotional exhaustion and job satisfaction. Practical and theoretical implications and directions for future research are also provided

The Optimal Noise in Noise-Contrastive Learning Is Not What You Think

Publisher Copyright: © 2022 Proceedings of the 38th Conference on Uncertainty in Artificial Intelligence, UAI 2022. All right reserved.Learning a parametric model of a data distribution is a well-known statistical problem that has seen renewed interest as it is brought to scale in deep learning. Framing the problem as a self-supervised task, where data samples are discriminated from noise samples, is at the core of state-of-the-art methods, beginning with Noise-Contrastive Estimation (NCE). Yet, such contrastive learning requires a good noise distribution, which is hard to specify; domain-specific heuristics are therefore widely used. While a comprehensive theory is missing, it is widely assumed that the optimal noise should in practice be made equal to the data, both in distribution and proportion; this setting underlies Generative Adversarial Networks (GANs) in particular. Here, we empirically and theoretically challenge this assumption on the optimal noise. We show that deviating from this assumption can actually lead to better statistical estimators, in terms of asymptotic variance. In particular, the optimal noise distribution is different from the data's and even from a different family.Peer reviewe

Impact of TV dramas on consumers' travel, shopping and purchase intentions

Countries are increasingly competing with each other to attract tourists. However, little is known about how consumers' tourism‐related behaviors respond to country image endeavors, such as TV dramas. We propose that as an important image source TV dramas from a country contribute to crafting the country's brand image and thereby influence viewers' tourism related intentions. Considering the case of Turkish TV dramas and drawing on the Hierarchical Decision Model (HDM), a survey of 400 Arab viewers revealed that exposure to Turkish TV dramas results in increased intentions to visit and shop in Turkey as well as to purchase products made in Turkey. Furthermore, while increase in purchase intentions results primarily from enhanced experiential associations related to the country, increase in visit or shopping intentions results from improved status associations related to the country

Deep Recurrent Encoder: an end-to-end network to model magnetoencephalography at scale

International audienc