Convergence of SGD for Training Neural Networks with Sliced Wasserstein Losses

Optimal Transport has sparked vivid interest in recent years, in particular thanks to the Wasserstein distance, which provides a geometrically sensible and intuitive way of comparing probability measures. For computational reasons, the Sliced Wasserstein (SW) distance was introduced as an alternative to the Wasserstein distance, and has seen uses for training generative Neural Networks (NNs). While convergence of Stochastic Gradient Descent (SGD) has been observed practically in such a setting, there is to our knowledge no theoretical guarantee for this observation. Leveraging recent works on convergence of SGD on non-smooth and non-convex functions by Bianchi et al. (2022), we aim to bridge that knowledge gap, and provide a realistic context under which fixed-step SGD trajectories for the SW loss on NN parameters converge. More precisely, we show that the trajectories approach the set of (sub)-gradient flow equations as the step decreases. Under stricter assumptions, we show a much stronger convergence result for noised and projected SGD schemes, namely that the long-run limits of the trajectories approach a set of generalised critical points of the loss function

Properties of Discrete Sliced Wasserstein Losses

The Sliced Wasserstein (SW) distance has become a popular alternative to the Wasserstein distance for comparing probability measures. Widespread applications include image processing, domain adaptation and generative modelling, where it is common to optimise some parameters in order to minimise SW, which serves as a loss function between discrete probability measures (since measures admitting densities are numerically unattainable). All these optimisation problems bear the same sub-problem, which is minimising the Sliced Wasserstein energy. In this paper we study the properties of $\mathcal{E}: Y \longmapsto \mathrm{SW}_2^2(\gamma_Y, \gamma_Z)$, i.e. the SW distance between two uniform discrete measures with the same amount of points as a function of the support $Y \in \mathbb{R}^{n \times d}$ of one of the measures. We investigate the regularity and optimisation properties of this energy, as well as its Monte-Carlo approximation $\mathcal{E}_p$ (estimating the expectation in SW using only $p$ samples) and show convergence results on the critical points of $\mathcal{E}_p$ to those of $\mathcal{E}$, as well as an almost-sure uniform convergence. Finally, we show that in a certain sense, Stochastic Gradient Descent methods minimising $\mathcal{E}$ and $\mathcal{E}_p$ converge towards (Clarke) critical points of these energies

Constrained Approximate Optimal Transport Maps

We investigate finding a map g within a function class G that minimises an Optimal Transport (OT) cost between a target measure ν and the image by g of a source measure µ. This is relevant when an OT map from µ to ν does not exist or does not satisfy the desired constraints of G. We address existence and uniqueness for generic subclasses of L-Lipschitz functions, including gradients of (strongly) convex functions and typical Neural Networks. We explore a variant that approaches a transport plan, showing equivalence to a map problem in some cases. For the squared Euclidean cost, we propose alternating minimisation over a transport plan π and map g, with the optimisation over g being the L 2 projection on G of the barycentric mapping π. In dimension one, this global problem equates the L 2 projection of π * onto G for an OT plan π * between µ and ν, but this does not extend to higher dimensions. We introduce a simple kernel method to find g within a Reproducing Kernel Hilbert Space in the discrete case. Finally, we present numerical methods for L-Lipschitz gradients of -strongly convex potentials.</div

Properties of Discrete Sliced Wasserstein Losses

The Sliced Wasserstein (SW) distance has become a popular alternative to the Wasserstein distance for comparing probability measures. Widespread applications include image processing, domain adaptation and generative modelling, where it is common to optimise some parameters in order to minimise SW, which serves as a loss function between discrete probability measures (since measures admitting densities are numerically unattainable). All these optimisation problems bear the same sub-problem, which is minimising the Sliced Wasserstein energy. In this paper we study the properties of E : Y −→ SW 2 2 (γ Y , γ Z), i.e. the SW distance between two uniform discrete measures with the same amount of points as a function of the support Y ∈ R n×d of one of the measures. We investigate the regularity and optimisation properties of this energy, as well as its Monte-Carlo approximation E p (estimating the expectation in SW using only p samples) and show convergence results on the critical points of E p to those of E, as well as an almost-sure uniform convergence. Finally, we show that in a certain sense, Stochastic Gradient Descent methods minimising E and E p converge towards (Clarke) critical points of these energies

Properties of Discrete Sliced Wasserstein Losses

International audienceThe Sliced Wasserstein (SW) distance has become a popular alternative to the Wasserstein distance for comparing probability measures. Widespread applications include image processing, domain adaptation and generative modelling, where it is common to optimise some parameters in order to minimise SW, which serves as a loss function between discrete probability measures (since measures admitting densities are numerically unattainable). All these optimisation problems bear the same sub-problem, which is minimising the Sliced Wasserstein energy. In this paper we study the properties of E : Y −→ SW 2 2 (γ Y , γ Z), i.e. the SW distance between two uniform discrete measures with the same amount of points as a function of the support Y ∈ R n×d of one of the measures. We investigate the regularity and optimisation properties of this energy, as well as its Monte-Carlo approximation E p (estimating the expectation in SW using only p samples) and show convergence results on the critical points of E p to those of E, as well as an almost-sure uniform convergence. Finally, we show that in a certain sense, Stochastic Gradient Descent methods minimising E and E p converge towards (Clarke) critical points of these energies

Reconstructing discrete measures from projections. Consequences on the empirical Sliced Wasserstein Distance

This paper deals with the reconstruction of a discrete measure γ Z on R d from the knowledge of its pushforward measures P i #γ Z by linear applications P i : R d → R di (for instance projections onto subspaces). The measure γ Z being fixed, assuming that the rows of the matrices P i are independent realizations of laws which do not give mass to hyperplanes, we show that if i d i > d, this reconstruction problem has almost certainly a unique solution. This holds for any number of points in γ Z. A direct consequence of this result is an almost-sure separability property on the empirical Sliced Wasserstein distance

Reconstructing discrete measures from projections. Consequences on the empirical Sliced Wasserstein Distance

This paper deals with the reconstruction of a discrete measure γ Z on R d from the knowledge of its pushforward measures P i #γ Z by linear applications P i : R d → R di (for instance projections onto subspaces). The measure γ Z being fixed, assuming that the rows of the matrices P i are independent realizations of laws which do not give mass to hyperplanes, we show that if i d i > d, this reconstruction problem has almost certainly a unique solution. This holds for any number of points in γ Z. A direct consequence of this result is an almost-sure separability property on the empirical Sliced Wasserstein distance

Nitrites, Nitrates, and Cardiovascular Outcomes: Are We Living “La Vie en Rose” With Pink Processed Meats?

International audienceBackground Nitrates and nitrites are used as food additives in processed meats. They are also commonly ingested from water and several foods. Several short‐term clinical studies suggested beneficial effects of dietary nitrates on blood pressure, while deleterious effects on oxidative damage have been suggested in some experimental studies. However, there is a lack of evidence from longitudinal epidemiological studies linking foods and water‐originated and additives‐originated nitrites and nitrates, separately, to hypertension and cardiovascular diseases risk. We aimed to study these associations in a large population‐based cohort. Methods and Results Overall, 106 288 adults from the French NutriNet‐Santé cohort (2009–2022) were included. Associations between nitrites and nitrates intakes and hypertension and cardiovascular disease risks were assessed using multi‐adjusted Cox proportional hazard models. During follow‐up, 3810 incident cases of hypertension and 2075 cases of cardiovascular diseases were ascertained. Participants with higher intakes of additives‐originated nitrites (sodium nitrite in particular [European code e250]) had a higher hypertension risk compared with nonconsumers (hazard ratio, 1.19 [95% CI, 1.08–1.32], P =0.001, and 1.19 [95% CI, 1.08–1.32], P =0.002), respectively. No association was detected between foods and water‐originated nitrites, or nitrates with hypertension risk (all P values >0.3). We found no association between nitrites or nitrates and risks of cardiovascular diseases (all P values >0.2). Conclusions These results do not support a protective role of nitrites or nitrates in cardiovascular health. Instead, they suggest a positive association between nitrites from food additives and hypertension risk, which needs confirmation in other large‐scale studies. These findings provide new evidence in the context of current discussions about updating regulations on the use of nitrites as food additives

Nitrites and nitrates from food additives and cancer risk: results from the NutriNet-Sante cohort

International audienceNitrates and nitrites occur naturally in water and soil and are commonly ingested from water and dietary sources. They are also frequently used as food additives mainly in processed meats. Experimental data consistently suggest their involvement in carcinogenesis but human data is still limited. The aim was to investigate the relationship between nitrate and nitrite intakes and the risk of cancer in a large prospective cohort with detailed and up-to-date dietary assessment. Overall, 101,056 adults from the French NutriNet-Santé cohort study (2009-ongoing) were included. Consumption of nitrites and nitrates was evaluated using repeated 24h dietary records, linked to a comprehensive food composition database which includes details of commercial names/brands of industrial products. Prospective associations between nitrite and nitrate exposures and the risk of cancer were assessed by multivariable Cox hazard models. During follow-up, 3311 first incident cancer cases were diagnosed. Compared with non-consumers, higher consumers of nitrates as food additives had higher risk of breast cancer (HR = 1.24 (1.03-1.48), P = 0.02); this was more specifically observed for potassium nitrate e252, P = 0.01). Higher consumers of nitrites as food additives, and specifically for sodium nitrite (e250), had a higher risk of prostate cancer (HR = 1.58 (1.14-2.18), P = 0.008 and HR = 1.62 (1.17-2.25), P = 0.004, respectively). No significant association was observed for nitrates and nitrites from natural sources. In this large prospective cohort, nitrates as food additives were positively associated with breast cancer risk and nitrites as food additives were positively associated with prostate cancer risk. While these results need confirmation in other large-scale prospective studies, they provide new insights in a context of lively debate around the ban of nitrite additives in food products