Sparse Continuous Distributions and Fenchel-Young Losses

Exponential families are widely used in machine learning, including many distributions in continuous and discrete domains (e.g., Gaussian, Dirichlet, Poisson, and categorical distributions via the softmax transformation). Distributions in each of these families have fixed support. In contrast, for finite domains, recent work on sparse alternatives to softmax (e.g., sparsemax, $\alpha$-entmax, and fusedmax), has led to distributions with varying support. This paper develops sparse alternatives to continuous distributions, based on several technical contributions: First, we define $\Omega$-regularized prediction maps and Fenchel-Young losses for arbitrary domains (possibly countably infinite or continuous). For linearly parametrized families, we show that minimization of Fenchel-Young losses is equivalent to moment matching of the statistics, generalizing a fundamental property of exponential families. When $\Omega$ is a Tsallis negentropy with parameter $\alpha$, we obtain ``deformed exponential families,'' which include $\alpha$-entmax and sparsemax ($\alpha=2$) as particular cases. For quadratic energy functions, the resulting densities are $\beta$-Gaussians, an instance of elliptical distributions that contain as particular cases the Gaussian, biweight, triweight, and Epanechnikov densities, and for which we derive closed-form expressions for the variance, Tsallis entropy, and Fenchel-Young loss. When $\Omega$ is a total variation or Sobolev regularizer, we obtain a continuous version of the fusedmax. Finally, we introduce continuous-domain attention mechanisms, deriving efficient gradient backpropagation algorithms for $\alpha \in \{1, 4/3, 3/2, 2\}$. Using these algorithms, we demonstrate our sparse continuous distributions for attention-based audio classification and visual question answering, showing that they allow attending to time intervals and compact regions.Comment: JMLR 2022 camera ready version. arXiv admin note: text overlap with arXiv:2006.0721

Efficient Methods for Natural Language Processing: A Survey

Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data; however, using only scale to improve performance means that resource consumption also grows. Such resources include data, time, storage, or energy, all of which are naturally limited and unevenly distributed. This motivates research into efficient methods that require fewer resources to achieve similar results. This survey synthesizes and relates current methods and findings in efficient NLP. We aim to provide both guidance for conducting NLP under limited resources, and point towards promising research directions for developing more efficient methods.Comment: Accepted at TACL, pre publication versio

Low in‑hospital mortality rate in patients with COVID‑19 receiving thromboprophylaxis: data from the multicentre observational START‑COVID Register

Abstract COVID-19 infection causes respiratory pathology with severe interstitial pneumonia and extra-pulmonary complications; in particular, it may predispose to thromboembolic disease. The current guidelines recommend the use of thromboprophylaxis in patients with COVID-19, however, the optimal heparin dosage treatment is not well-established. We conducted a multicentre, Italian, retrospective, observational study on COVID-19 patients admitted to ordinary wards, to describe clinical characteristic of patients at admission, bleeding and thrombotic events occurring during hospital stay. The strategies used for thromboprophylaxis and its role on patient outcome were, also, described. 1091 patients hospitalized were included in the START-COVID-19 Register. During hospital stay, 769 (70.7%) patients were treated with antithrombotic drugs: low molecular weight heparin (the great majority enoxaparin), fondaparinux, or unfractioned heparin. These patients were more frequently affected by comorbidities, such as hypertension, atrial fibrillation, previous thromboembolism, neurological disease,and cancer with respect to patients who did not receive thromboprophylaxis. During hospital stay, 1.2% patients had a major bleeding event. All patients were treated with antithrombotic drugs; 5.4%, had venous thromboembolism [30.5% deep vein thrombosis (DVT), 66.1% pulmonary embolism (PE), and 3.4% patients had DVT + PE]. In our cohort the mortality rate was 18.3%. Heparin use was independently associated with survival in patients aged ≥ 59 years at multivariable analysis. We confirmed the high mortality rate of COVID-19 in hospitalized patients in ordinary wards. Treatment with antithrombotic drugs is significantly associated with a reduction of mortality rates especially in patients older than 59 years