Convergence of Proximal Point and Extragradient-Based Methods Beyond Monotonicity: the Case of Negative Comonotonicity

Algorithms for min-max optimization and variational inequalities are often studied under monotonicity assumptions. Motivated by non-monotone machine learning applications, we follow the line of works [Diakonikolas et al., 2021, Lee and Kim, 2021, Pethick et al., 2022, B\"ohm, 2022] aiming at going beyond monotonicity by considering the weaker negative comonotonicity assumption. In particular, we provide tight complexity analyses for the Proximal Point, Extragradient, and Optimistic Gradient methods in this setup, closing some questions on their working guarantees beyond monotonicity.Comment: 34 pages, 2 figures. Changes in V2: missing reference was added. Code: https://github.com/eduardgorbunov/Proximal_Point_and_Extragradient_based_methods_negative_comonotonicit

Variance Reduction is an Antidote to Byzantines: Better Rates, Weaker Assumptions and Communication Compression as a Cherry on the Top

Byzantine-robustness has been gaining a lot of attention due to the growth of the interest in collaborative and federated learning. However, many fruitful directions, such as the usage of variance reduction for achieving robustness and communication compression for reducing communication costs, remain weakly explored in the field. This work addresses this gap and proposes Byz-VR-MARINA - a new Byzantine-tolerant method with variance reduction and compression. A key message of our paper is that variance reduction is key to fighting Byzantine workers more effectively. At the same time, communication compression is a bonus that makes the process more communication efficient. We derive theoretical convergence guarantees for Byz-VR-MARINA outperforming previous state-of-the-art for general non-convex and Polyak-Lojasiewicz loss functions. Unlike the concurrent Byzantine-robust methods with variance reduction and/or compression, our complexity results are tight and do not rely on restrictive assumptions such as boundedness of the gradients or limited compression. Moreover, we provide the first analysis of a Byzantine-tolerant method supporting non-uniform sampling of stochastic gradients. Numerical experiments corroborate our theoretical findings.Comment: 41 pages, 6 figures. Changes in v2: few typos and inaccuracies were fixed, more clarifications were added. Code: https://github.com/SamuelHorvath/VR_Byzantin

High-probability bounds for stochastic optimization and variational inequalities: The case of unbounded variance

During recent years the interest of optimization and machine learning communities in high-probability convergence of stochastic optimization methods has been growing. One of the main reasons for this is that high-probability complexity bounds are more accurate and less studied than in-expectation ones. However, SOTA high-probability non-asymptotic convergence results are derived under strong assumptions such as the boundedness of the gradient noise variance or of the objective's gradient itself. In this paper, we propose several algorithms with high-probability convergence results under less restrictive assumptions. In particular, we derive new high-probability convergence results under the assumption that the gradient/operator noise has bounded central α-th moment for α∈(1,2] in the following setups: (i) smooth non-convex / Polyak-Lojasiewicz / convex / strongly convex / quasi-strongly convex minimization problems, (ii) Lipschitz / star-cocoercive and monotone / quasi-strongly monotone variational inequalities. These results justify the usage of the considered methods for solving problems that do not fit standard functional classes studied in stochastic optimization

High-Probability Bounds for Stochastic Optimization and Variational Inequalities: the Case of Unbounded Variance

During recent years the interest of optimization and machine learning communities in high-probability convergence of stochastic optimization methods has been growing. One of the main reasons for this is that high-probability complexity bounds are more accurate and less studied than in-expectation ones. However, SOTA high-probability non-asymptotic convergence results are derived under strong assumptions such as the boundedness of the gradient noise variance or of the objective's gradient itself. In this paper, we propose several algorithms with high-probability convergence results under less restrictive assumptions. In particular, we derive new high-probability convergence results under the assumption that the gradient/operator noise has bounded central $\alpha$-th moment for $\alpha \in (1,2]$ in the following setups: (i) smooth non-convex / Polyak-Lojasiewicz / convex / strongly convex / quasi-strongly convex minimization problems, (ii) Lipschitz / star-cocoercive and monotone / quasi-strongly monotone variational inequalities. These results justify the usage of the considered methods for solving problems that do not fit standard functional classes studied in stochastic optimization.Comment: ICML 2023. 86 pages. Changes in v2: ICML formatting was applied along with minor edits of the tex

Méthode de détection et de suivi multi-piétons multi-capteurs embarquée sur un véhicule routier: application à un environnement urbain

At first my work consisted in developing an appropriate method to detect, then identify and track pedestrians in an outdoor environment from a single four-plane laser sensor. Pedestrian extraction and the merging of the four laser planes are both based on a nonparametric kernel method, also called "Parzen Windows". Initially, this estimator is used to approximate the likelihood function of the impact record in the laser image according to a pedestrian's geometrical characteristics. Secondly this estimator is used to calculate the likelihood that a pedestrian should be located within the four laser planes. Finally, to best characterize the complex trajectory of a pedestrian, the tracking process is based on the traditional particle filter. Unfortunately a pedestrian detection system which relies only on a laser sensor remains unsatisfactory as far as performance is concerned. Indeed, the inherent limitations of this sensor (no information about height, the outline or the color of the objects), as well as its sensitivity to such atmospheric conditions as rain or fog, make it necessary to resort to a multisensorial solution which allows to effectively combine the information provided by the laser and video sensors. This fusion methods is based on the development of a non-parametric method for data association, which allows to keep all the information contained in the measurements sent by the laser and video sensors. The performance of each proposed algorithm was characterized and reviewed, using real data obtained from numerous recording on board the LASMEA and Renault test vehicle ; Renault being the French vehicle manufacturer with whom we collaborate on our ANR LOVe project.Les travaux présentés dans cette thèse ont pour cadre la vision par ordinateur et concernent la détection et le suivi de piéton se trouvant sur la trajectoire d'un véhicule routier circulant en milieu urbain. Dans ce type d'environnement complexe, une des difficultés majeurs est la capacité à discerner les piétons des nombreux autres obstacles situés sur la chaussée. Un autre point essentiel est de pouvoir les suivre afin de prédire leur déplacement et ainsi le cas échéant éviter le contact avec le véhicule. D'autres contraintes s'ajoutent dans le contexte industriel des véhicules routiers intelligents. Il est nécessaire de proposer des algorithmes robustes temps réel avec des capteurs les moins chers possibl

Méthode de détection et de suivi multi-piétons multi-capteurs embarquée sur un véhicule routier (application à un environnement urbain)

Les travaux présentés dans cette thèse ont pour cadre la vision par ordinateur et concernent la détection et le suivi de piéton se trouvant sur la trajectoire d'un véhicule routier circulant en milieu urbain. Dans ce type d'environnement complexe, une des difficultés majeurs est la capacité à discerner les piétons des nombreux autres obstacles situés sur la chaussée. Un autre point essentiel est de pouvoir les suivre afin de prédire leur déplacement et ainsi le cas échéant éviter le contact avec le véhicule. D'autres contraintes s'ajoutent dans le contexte industriel des véhicules routiers intelligents. Il est nécessaire de proposer des algorithmes robustes temps réel avec des capteurs les moins chers possibleCLERMONT FD-BCIU Sci.et Tech. (630142101) / SudocSudocFranceF

Système de détection de piétons à bord de véhicules : approche par télémètrie laser

National audienceCet article traite de la détection des piétons à l'aide d'un capteur laser. Ce capteur placé à l'avant d'un véhicule recueille des informations distance réparties selon quatre nappes. Au même titre qu'un véhicule, un piéton constitue dans un environnement de conduite un obstacle qu'il faut détecter, localiser puis identifier et éventuellement suivre. Une méthode de détection, d'identification puis de suivi de piétons à partir de cet unique capteur laser quatre plans est proposée et discutée. Afin d'isoler les piétons, une méthode non-paramétrique exploitant la technique de fenêtrage de Parzen est présentée. Le pistage repose sur l'utilisation classique d'un filtre à particules. Les résultats obtenus sur des données réelles sont fournis

Data Fusion Performance Evaluation for Range Measurements Combine with Cartesian ones for Road Obstacle Tracking

International audienceThis paper deals with evaluation of centralized fusion for two dissimilar sensors for the purpose of road obstacle tracking. The aim of sensor fusion is to produce an improved state estimate of a system from a set of independent data sources. Indeed, for a robust environment perception, see as obstacles here, several sensors should be installed in the equipped vehicle: camera, lidar, radar, etc. In our case, the motivation for this work comes from the need to track road targets with lidar measurements combined to radar ones. Thus, the aim is to combine effectively radar range measurements (i.e. range and range rate) with Lidar Cartesian measurements for a ”turn” scenario. Centralized fusion, i.e. measurement fusion, for two dissimilar sensors is considered here for evaluation. Evaluation is based on Cramer-Rao Lower Bound (CRLB) which is the basic tool for investigating estimation performance as it represents a limit of cognizability of the state. In the target tracking area, a recursive formulation of the Posterior Cramer- Rao Lower Bound (PCRLB) is used to analyze performance. Many bound comparisons are made according to used scenarios and various sensors configurations. Moreover, two algorithms for target motion analysis are developed and compared to the theoretical bounds of performance: the extended Kalman filter and the particle filter

Nonparametric data association for particle filter based multi-object tracking : application to multi-pedestrian tracking

International audienceThis article deals with the following issue: how to track a varying number of pedestrians through observations by means of a 4-plane laser sensor. In order to answer to the multiple target tracking problem and more specifically pedestrian tracking, we propose in this paper a statistical approach using a particle filter based on nonparametric data association methods. This approach allows to go beyond the conventional Gaussian assumption and to use as well as possible each particle during track/observation association by means of either a "Parzen Window" kernel method or a K-nearest neighbor algorithm. Simulated and experimental results show the relevance of this method compared to the usual Gaussian window methods