Bundle methods for regularized risk minimization with applications to robust learning

Supervised learning in general and regularized risk minimization in particular is about solving optimization problem which is jointly defined by a performance measure and a set of labeled training examples. The outcome of learning, a model, is then used mainly for predicting the labels for unlabeled examples in the testing environment. In real-world scenarios: a typical learning process often involves solving a sequence of similar problems with different parameters before a final model is identified. For learning to be successful, the final model must be produced timely, and the model should be robust to (mild) irregularities in the testing environment. The purpose of this thesis is to investigate ways to speed up the learning process and improve the robustness of the learned model. We first develop a batch convex optimization solver specialized to the regularized risk minimization based on standard bundle methods. The solver inherits two main properties of the standard bundle methods. Firstly, it is capable of solving both differentiable and non-differentiable problems, hence its implementation can be reused for different tasks with minimal modification. Secondly, the optimization is easily amenable to parallel and distributed computation settings; this makes the solver highly scalable in the number of training examples. However, unlike the standard bundle methods, the solver does not have extra parameters which need careful tuning. Furthermore, we prove that the solver has faster convergence rate. In addition to that, the solver is very efficient in computing approximate regularization path and model selection. We also present a convex risk formulation for incorporating invariances and prior knowledge into the learning problem. This formulation generalizes many existing approaches for robust learning in the setting of insufficient or noisy training examples and covariate shift. Lastly, we extend a non-convex risk formulation for binary classification to structured prediction. Empirical results show that the model obtained with this risk formulation is robust to outliers in the training examples

Machine Unlearning: A Survey

Machine learning has attracted widespread attention and evolved into an enabling technology for a wide range of highly successful applications, such as intelligent computer vision, speech recognition, medical diagnosis, and more. Yet a special need has arisen where, due to privacy, usability, and/or the right to be forgotten, information about some specific samples needs to be removed from a model, called machine unlearning. This emerging technology has drawn significant interest from both academics and industry due to its innovation and practicality. At the same time, this ambitious problem has led to numerous research efforts aimed at confronting its challenges. To the best of our knowledge, no study has analyzed this complex topic or compared the feasibility of existing unlearning solutions in different kinds of scenarios. Accordingly, with this survey, we aim to capture the key concepts of unlearning techniques. The existing solutions are classified and summarized based on their characteristics within an up-to-date and comprehensive review of each category's advantages and limitations. The survey concludes by highlighting some of the outstanding issues with unlearning techniques, along with some feasible directions for new research opportunities

Conditional Gradient Methods

The purpose of this survey is to serve both as a gentle introduction and a coherent overview of state-of-the-art Frank--Wolfe algorithms, also called conditional gradient algorithms, for function minimization. These algorithms are especially useful in convex optimization when linear optimization is cheaper than projections. The selection of the material has been guided by the principle of highlighting crucial ideas as well as presenting new approaches that we believe might become important in the future, with ample citations even of old works imperative in the development of newer methods. Yet, our selection is sometimes biased, and need not reflect consensus of the research community, and we have certainly missed recent important contributions. After all the research area of Frank--Wolfe is very active, making it a moving target. We apologize sincerely in advance for any such distortions and we fully acknowledge: We stand on the shoulder of giants.Comment: 238 pages with many figures. The FrankWolfe.jl Julia package (https://github.com/ZIB-IOL/FrankWolfe.jl) providces state-of-the-art implementations of many Frank--Wolfe method

Generalized averaged Gaussian quadrature and applications

A simple numerical method for constructing the optimal generalized averaged Gaussian quadrature formulas will be presented. These formulas exist in many cases in which real positive GaussKronrod formulas do not exist, and can be used as an adequate alternative in order to estimate the error of a Gaussian rule. We also investigate the conditions under which the optimal averaged Gaussian quadrature formulas and their truncated variants are internal

MS FT-2-2 7 Orthogonal polynomials and quadrature: Theory, computation, and applications

Quadrature rules find many applications in science and engineering. Their analysis is a classical area of applied mathematics and continues to attract considerable attention. This seminar brings together speakers with expertise in a large variety of quadrature rules. It is the aim of the seminar to provide an overview of recent developments in the analysis of quadrature rules. The computation of error estimates and novel applications also are described

Decentralization in the Kyrgyz agricultural sector

Seit der Unabhängigkeit der zentralasiatischen Republik Kirgisistan haben Politik, Verwaltung und Ökonomie verschiedene Formen von Dezentralisierung erfahren. Diese Dissertation umfasst fünf Essays, die die Dezentralisierung im landwirtschafltichen Sektor aus institutionenökonomischer Sicht untersuchen. Die ersten zwei Essays geben detaillierte Einblicke in die institutionellen Rahmenbedingungen von Dezentralisierung und beurteilen ihrer Wirkung in Hinblick auf Serviceverfügbarkeit und -qualität in dörflichen Gemeinden. Die folgenden drei Essays untersuchen, anhand einzelner und multipler Fallstudien, ein spezifisches Beispiel der Dezentralisierung landwirtschaftlicher Services: die Einführung von gemeindebasiertem Weidemanagement. Es lassen sich drei Ergebnisse ableiten: Erstens, internationale Nichtregierungsorganisationen (NROs) steuern das ländliche Dienstleistungsangebot und fördern die Bildung gemeindebasierter Nutzergruppen für ausgewählte Services. Zweitens, Institutionen zur Implementierung der Servicebereitstellung werden von NROs entwickelt; drittens, die Servicebereitsstellung ist nicht befriedigend und das Potential zur Berücksichtigung lokaler Servicebedürfnisse und lokalen Wissens wird nur teilweise ausgeschöpft, da die Implementierung keine umfassende Servicenutzerbeteiligung sicherstellt. Die Wirkungen gemeindebasierter Dezentralisierungsprozesse sind als Ergebnis rationaler Handlungsentscheidungen von lokalen Mitarbeitern der NRO und Verantwortlichen in der dörflichen Verwaltung zu verstehen. Diese Entscheidungen sind vielfach durch extern entwickelte, und teilweise unpassende, Institutionen bestimmt. Verbesserte Implementierungsstrategien sind daher notwendig. Diese sind auf Basis detaillierter qualitativer Studien des lokalen Umsetzungskontexts zu entwickeln.Since the Central Asian Kyrgyz Republic gained independence from the Soviet Union, policy making, administration and economy have seen some form of decentralization. This dissertation contains five essays which study decentralization in the Kyrgyz agricultural sector from an institutional economics perspective. The first two essays provide in-depth information on the institutional setting of decentralization and its effects on service availability and quality at municipality level. The subsequent three essays explore, based on single and multiple case studies, one specific field of decentralized agricultural services: a community-based natural resource management reform in the pasture sector. The three key findings are: first, international NGOs govern rural service provision and support the creation of community-based service user groups for selected services; second, the NGOs design institutions for implementation and provide financial resources; third, service provision is unadequate and, because implementation does not provide for broader service user involvement in decision making, service user needs and local knowledge impact service decisions only to a very limited degree. The overall result of the dissertation is that the municipality-level processes of decentralization must be understood as outcomes of rational decision making of lowest-level NGO staff and municipality level policy administrators. These decisions are impacted by partly inappropriate, externally designed implementation institutions. Improved implementation rule design is therefore needed. The recommendation from this research is therefore to use detailed qualitative studies of implementation contexts as a basis for developing better tailored implementation strategies