An Introduction to Bi-level Optimization: Foundations and Applications in Signal Processing and Machine Learning

Recently, bi-level optimization (BLO) has taken center stage in some very exciting developments in the area of signal processing (SP) and machine learning (ML). Roughly speaking, BLO is a classical optimization problem that involves two levels of hierarchy (i.e., upper and lower levels), wherein obtaining the solution to the upper-level problem requires solving the lower-level one. BLO has become popular largely because it is powerful in modeling problems in SP and ML, among others, that involve optimizing nested objective functions. Prominent applications of BLO range from resource allocation for wireless systems to adversarial machine learning. In this work, we focus on a class of tractable BLO problems that often appear in SP and ML applications. We provide an overview of some basic concepts of this class of BLO problems, such as their optimality conditions, standard algorithms (including their optimization principles and practical implementations), as well as how they can be leveraged to obtain state-of-the-art results for a number of key SP and ML applications. Further, we discuss some recent advances in BLO theory, its implications for applications, and point out some limitations of the state-of-the-art that require significant future research efforts. Overall, we hope that this article can serve to accelerate the adoption of BLO as a generic tool to model, analyze, and innovate on a wide array of emerging SP and ML applications

Vertical Profiles of Aerosol Optical and Microphysical Properties During a Rare Case of Long-range Transport of Mixed Biomass Burning-polluted Dust Aerosols from the Russian Federation-kazakhstan to Athens, Greece

Multi-wavelength aerosol Raman lidar measurements with elastic depolarization at 532 nm were combined with sun photometry during the HYGRA-CD campaign over Athens, Greece, on May-June 2014. We retrieved the aerosol optical [3 aerosol backscatter profiles (baer) at 355-532-1064 nm, 2 aerosol extinction (aaer) profiles at 355-532 nm and the aerosol linear depolarization ratio (δ) at 532 nm] and microphysical properties [effective radius (reff), complex refractive index (m), single scattering albedo (ω)]. We present a case study of a long distance transport (~3.500-4.000 km) of biomass burning particles mixed with dust from the Russian Federation-Kazakhstan regions arriving over Athens on 21-23 May 2014 (1.7-3.5 km height). On 23 May, between 2-2.75 km we measured mean lidar ratios (LR) of 35 sr (355 nm) and 42 sr (532 nm), while the mean Ångström exponent (AE) aerosol backscatter-related values (355nm/532nm and 532nm/1064nm) were 2.05 and 1.22, respectively; the mean value of δ at 532 nm was measured to be 9%. For that day the retrieved mean aerosol microphysical properties at 2-2.75 km height were: reff=0.26 μm (fine mode), reff=2.15 μm (coarse mode), m=1.36+0.00024i, ω=0.999 (355 nm, fine mode), ω=0.992(355 nm, coarse mode), ω=0.997 (532 nm, fine mode), and ω=0.980 (532 nm, coarse mode)

Minimax Problems with Coupled Linear Constraints: Computational Complexity, Duality and Solution Methods

In this work we study a special minimax problem where there are linear constraints that couple both the minimization and maximization decision variables. The problem is a generalization of the traditional saddle point problem (which does not have the coupling constraint), and it finds applications in wireless communication, game theory, transportation, just to name a few. We show that the considered problem is challenging, in the sense that it violates the classical max-min inequality, and that it is NP-hard even under very strong assumptions (e.g., when the objective is strongly convex-strongly concave). We then develop a duality theory for it, and analyze conditions under which the duality gap becomes zero. Finally, we study a class of stationary solutions defined based on the dual problem, and evaluate their practical performance in an application on adversarial attacks on network flow problems

EXPLOITING INTERNET & MULTIMEDIA TECHNOLOGIES TO ADVANCE EDUCATIONAL PROCESS

Internet technologies and multimedia applications are becoming the new trend in educational process. The number of schools, universities and other educational institutions that intend to adopt computerbased learning is being considerably increased. Thus there are some serious aspects to be discussed on this area. In our paper we present techniques incorporated in the organized design and the augmentative development of an adaptive web-based educational environment. We have included personalization characteristics in the educational objects and supportive communication services. This way the educational process is customized to the learning curve, the standards and the preferences of each user. Even more the opportunity to support virtual classrooms and tele-conference operations is given to the user. Due to the non-centralized character of the system such distributed actions are feasible to be done. Representative feature of our system is the application of innovative and well-known user interface and multimedia approaches where the latest and most modern technologies on the specific field are being considered and thoroughly used through the system’s interaction components. In this direction we have paid a lot of effort to control and enhance performance matters. The successful use of any web environment depends on its high-quality performance and availability even under the most aggravating circumstances. A combined web engineering methodology has been adopted for the theoretical and practical model standardization. The main goal of this is to support scaling, reusability and maintenance of the webbased learning environment both in users ’ terms and educational material so as to cover the fields of intelligence and adaptiveness

Zeroth-Order SciML: Non-intrusive Integration of Scientific Software with Deep Learning

Using deep learning (DL) to accelerate and/or improve scientific workflows can yield discoveries that are otherwise impossible. Unfortunately, DL models have yielded limited success in complex scientific domains due to large data requirements. In this work, we propose to overcome this issue by integrating the abundance of scientific knowledge sources (SKS) with the DL training process. Existing knowledge integration approaches are limited to using differentiable knowledge source to be compatible with first-order DL training paradigm. In contrast, our proposed approach treats knowledge source as a black-box in turn allowing to integrate virtually any knowledge source. To enable an end-to-end training of SKS-coupled-DL, we propose to use zeroth-order optimization (ZOO) based gradient-free training schemes, which is non-intrusive, i.e., does not require making any changes to the SKS. We evaluate the performance of our ZOO training scheme on two real-world material science applications. We show that proposed scheme is able to effectively integrate scientific knowledge with DL training and is able to outperform purely data-driven model for data-limited scientific applications. We also discuss some limitations of the proposed method and mention potentially worthwhile future directions

Tropospheric Vertical Profiles of Aerosol Optical, Microphysical and Concentration Properties in the Frame of the Hygra-CD Campaign (Athens, Greece 2014): A Case Study of Long-Range Transport of Mixed Aerosols

Combined multi-wavelength aerosol Raman lidar and sun photometry measurements were performed during the HYGRA-CD campaign over Athens, Greece during May-June 2014. The retrieved aerosol optical properties (3 aerosol backscatter at 355-532-1064 nm and 2 aerosol extinction profiles at 355-532 nm) were used as input to an inversion code to retrieve the aerosol microphysical properties (effective radius reff and number concentration N) using regularization techniques. Additionally, the volume concentration profile was derived for fine particles using the LIRIC code. In this paper we selected a complex case study of long-range transport of mixed aerosols (biomass burning particles mixed with dust) arriving over Athens between 10-12 June 2014 in the 1.5-4 km height. Between 2-3 km height we measured mean lidar ratios (LR) ranging from 45 to 58 sr (at 355 and 532 nm), while the Ångström exponent (AE) aerosol extinction-related values (355nm/532nm) ranged between 0.8-1.3. The retrieved values of reff and N ranged from 0.19±0.07 to 0.22±0.07 μm and 460±230 to 2200±2800 cm-3, respectively. The aerosol linear depolarization ratio (δ) at 532 nm was lower than 5-7% (except for the Saharan dust cases, where δ~10-15%)