Towards Out-of-Distribution Adversarial Robustness

Adversarial robustness continues to be a major challenge for deep learning. A core issue is that robustness to one type of attack often fails to transfer to other attacks. While prior work establishes a theoretical trade-off in robustness against different $L_p$ norms, we show that there is potential for improvement against many commonly used attacks by adopting a domain generalisation approach. Concretely, we treat each type of attack as a domain, and apply the Risk Extrapolation method (REx), which promotes similar levels of robustness against all training attacks. Compared to existing methods, we obtain similar or superior worst-case adversarial robustness on attacks seen during training. Moreover, we achieve superior performance on families or tunings of attacks only encountered at test time. On ensembles of attacks, our approach improves the accuracy from 3.4% the best existing baseline to 25.9% on MNIST, and from 16.9% to 23.5% on CIFAR10.Comment: Under review ICLR 202

Continual Pre-Training of Large Language Models: How to (re)warm your model?

Large language models (LLMs) are routinely pre-trained on billions of tokens, only to restart the process over again once new data becomes available. A much cheaper and more efficient solution would be to enable the continual pre-training of these models, i.e. updating pre-trained models with new data instead of re-training them from scratch. However, the distribution shift induced by novel data typically results in degraded performance on past data. Taking a step towards efficient continual pre-training, in this work, we examine the effect of different warm-up strategies. Our hypothesis is that the learning rate must be re-increased to improve compute efficiency when training on a new dataset. We study the warmup phase of models pre-trained on the Pile (upstream data, 300B tokens) as we continue to pre-train on SlimPajama (downstream data, 297B tokens), following a linear warmup and cosine decay schedule. We conduct all experiments on the Pythia 410M language model architecture and evaluate performance through validation perplexity. We experiment with different pre-training checkpoints, various maximum learning rates, and various warmup lengths. Our results show that while rewarming models first increases the loss on upstream and downstream data, in the longer run it improves the downstream performance, outperforming models trained from scratch\unicode{x2013}even for a large downstream dataset

A novel Big Data analytics and intelligent technique to predict driver's intent

Modern age offers a great potential for automatically predicting the driver's intent through the increasing miniaturization of computing technologies, rapid advancements in communication technologies and continuous connectivity of heterogeneous smart objects. Inside the cabin and engine of modern cars, dedicated computer systems need to possess the ability to exploit the wealth of information generated by heterogeneous data sources with different contextual and conceptual representations. Processing and utilizing this diverse and voluminous data, involves many challenges concerning the design of the computational technique used to perform this task. In this paper, we investigate the various data sources available in the car and the surrounding environment, which can be utilized as inputs in order to predict driver's intent and behavior. As part of investigating these potential data sources, we conducted experiments on e-calendars for a large number of employees, and have reviewed a number of available geo referencing systems. Through the results of a statistical analysis and by computing location recognition accuracy results, we explored in detail the potential utilization of calendar location data to detect the driver's intentions. In order to exploit the numerous diverse data inputs available in modern vehicles, we investigate the suitability of different Computational Intelligence (CI) techniques, and propose a novel fuzzy computational modelling methodology. Finally, we outline the impact of applying advanced CI and Big Data analytics techniques in modern vehicles on the driver and society in general, and discuss ethical and legal issues arising from the deployment of intelligent self-learning cars

Upstream control strategy development for afucosylated species in mAb biomanufacturing

Please click Additional Files below to see the full abstract

Refractive index measurement of pharmaceutical powders in the short-wave infrared range using index matching assisted with phase imaging

Refractive index is a fundamental optical property of powder and a key input to the measurement of the size distribution using light scattering and the measurement of the absorption and scattering coefficients using diffuse reflectance spectroscopy. In this work, we demonstrate a method that combines index matching with phase imaging to determine the refractive index of complex powder particles at high accuracy without relying on any assumptions. Index matching is robust, but the accuracy is not high by relying on ocular observation of the interface. Phase imaging using interferometry improves the accuracy of determining the match by an order of magnitude. Using the proposed method, we determine the refractive index of four types of pharmaceutical powder in the short-wave infrared range of 1100-1650 nm

Identification of Cell Culture Factors Influencing Afucosylation Levels in Monoclonal Antibodies by Partial Least-Squares Regression and Variable Importance Metrics

Retrospective analysis of historic data for cell culture processes is a powerful tool to develop further process understanding. In particular, deploying retrospective analyses can identify important cell culture process parameters for controlling critical quality attributes, e.g., afucosylation, for the production of monoclonal antibodies (mAbs). However, a challenge of analyzing large cell culture data is the high correlation between regressors (particularly media composition), which makes traditional analyses, such as analysis of variance and multivariate linear regression, inappropriate. Instead, partial least-squares regression (PLSR) models, in combination with machine learning techniques such as variable importance metrics, are an orthogonal or alternative approach to identifying important regressors and overcoming the challenge of a highly covariant data structure. A specific workflow for the retrospective analysis of cell culture data is proposed that covers data curation, PLS regression, model analysis, and further steps. In this study, the proposed workflow was applied to data from four mAb products in an industrial cell culture process to identify significant process parameters that influence the afucosylation levels. The PLSR workflow successfully identified several significant parameters, such as temperature and media composition, to enhance process understanding of the relationship between cell culture processes and afucosylation levels

Observational and genetic associations between cardiorespiratory fitness and cancer:a UK Biobank and international consortia study

Background: The association of fitness with cancer risk is not clear. Methods: We used Cox proportional hazards models to estimate hazard ratios (HRs) and 95% confidence intervals (CIs) for risk of lung, colorectal, endometrial, breast, and prostate cancer in a subset of UK Biobank participants who completed a submaximal fitness test in 2009-12 (N = 72,572). We also investigated relationships using two-sample Mendelian randomisation (MR), odds ratios (ORs) were estimated using the inverse-variance weighted method.Results: After a median of 11 years of follow-up, 4290 cancers of interest were diagnosed. A 3.5 ml O2⋅min−1⋅kg−1 total-body mass increase in fitness (equivalent to 1 metabolic equivalent of task (MET), approximately 0.5 standard deviation (SD)) was associated with lower risks of endometrial (HR = 0.81, 95% CI: 0.73–0.89), colorectal (0.94, 0.90–0.99), and breast cancer (0.96, 0.92–0.99). In MR analyses, a 0.5 SD increase in genetically predicted O2⋅min−1⋅kg−1 fat-free mass was associated with a lower risk of breast cancer (OR = 0.92, 95% CI: 0.86–0.98). After adjusting for adiposity, both the observational and genetic associations were attenuated. Discussion: Higher fitness levels may reduce risks of endometrial, colorectal, and breast cancer, though relationships with adiposity are complex and may mediate these relationships. Increasing fitness, including via changes in body composition, may be an effective strategy for cancer prevention.</p