Meta-learning for Multi-variable Non-convex Optimization Problems: Iterating Non-optimums Makes Optimum Possible

In this paper, we aim to address the problem of solving a non-convex optimization problem over an intersection of multiple variable sets. This kind of problems is typically solved by using an alternating minimization (AM) strategy which splits the overall problem into a set of sub-problems corresponding to each variable, and then iteratively performs minimization over each sub-problem using a fixed updating rule. However, due to the intrinsic non-convexity of the overall problem, the optimization can usually be trapped into bad local minimum even when each sub-problem can be globally optimized at each iteration. To tackle this problem, we propose a meta-learning based Global Scope Optimization (GSO) method. It adaptively generates optimizers for sub-problems via meta-learners and constantly updates these meta-learners with respect to the global loss information of the overall problem. Therefore, the sub-problems are optimized with the objective of minimizing the global loss specifically. We evaluate the proposed model on a number of simulations, including solving bi-linear inverse problems: matrix completion, and non-linear problems: Gaussian mixture models. The experimental results show that our proposed approach outperforms AM-based methods in standard settings, and is able to achieve effective optimization in some challenging cases while other methods would typically fail.Comment: 15 pages, 8 figure

Semi-Supervised Learning of Hidden Markov Models via a Homotopy Method

Hidden Markov model (HMM) classifier design is considered for analysis of sequential data, incorporating both labeled and unlabeled data for training; the balance between labeled and unlabeled data is controlled by an allocation parameter lambda in [0, 1), where lambda = 0 corresponds to purely supervised HMM learning (based only on the labeled data) and lambda = 1 corresponds to unsupervised HMM-based clustering (based only on the unlabeled data). The associated estimation problem can typically be reduced to solving a set of fixed point equations in the form of a “natural-parameter homotopy”. This paper applies a homotopy method to track a continuous path of solutions, starting from a local supervised solution (lambda = 0) to a local unsupervised solution (lambda = 1). The homotopy method is guaranteed to track with probability one from lambda = 0 to lambda = 1 if the lambda = 0 solution is unique; this condition is not satisfied for the HMM, since the maximum likelihood supervised solution (lambda = 0) is characterized by many local optimal solutions. A modified form of the homotopy map for HMMs assures a track from lambda = 0 to lambda = 1. Following this track leads to a formulation for selecting lambda in [0, 1) for a semi-supervised solution, and it also provides a tool for selection from among multiple (local optimal) supervised solutions. The results of applying the proposed method to measured and synthetic sequential data verify its robustness and feasibility compared to the conventional EM approach for semi-supervised HMM training

Producing PID controllers for testing clustering - Investigating novelty detection for use in classifying PID parameters

PID controllers performance depend on how they are tuned. Tuning a controller is not easy either and many use their experience and intuition, or automatic software for tuning. We present a way to test the quality of controllers using statistics. The method uses multivariate extreme value statistics with novelty detection. With the analyser presented in this paper one can compare fresh PID parameters to those that have been tuned well. This tool can help in troubleshooting with PID controller tuning. Conventional novelty detection methods use a Gaussian mixture model, the analyser here uses a variational mixture model instead. This made the fitting process easier for the user. Part of this work was to create PID parameter configurations to test the analyser with. We needed both well tuned and poorly tuned parameters for testing the algorithm, as well as several examples of both cases. A genetic algorithm was seen as a tool that would meet these requirements. Genetic algorithms have previously been used for both test parameters generation and PID controller tuning in many applications. The genetic algorithm was written in Matlab. The reason for using Matlab is that the genetic algorithm uses a Simulink model of a PID control process in its fitness function. The parameters were simulated and plots of their step response were drawn. The best configurations according to the genetic algorithm had little error compared to the reference value. The error seemed to rise according to the index of goodness used by the genetic algorithm. We set three criterions on the parameters: maximum overshoot, settling time, and sum of absolute error. Each of these criterions had a threshold. Each parameter configuration that crossed at least one of these thresholds were classed abnormal. The performance of the analyser was assessed with these parameters. The analyser were first trained with a set of normal parameters, then tested with a set of normal and a set of abnormal parameters. The results showed 2 false alarms in both cases out of 104 possible. This gave us an accuracy of 98%, which is a very high one for a novelty detection method.fi=Opinnäytetyö kokotekstinä PDF-muodossa.|en=Thesis fulltext in PDF format.|sv=Lärdomsprov tillgängligt som fulltext i PDF-format

Exploration and Optimization of Noise Reduction Algorithms for Speech Recognition in Embedded Devices

Environmental noise present in real-life applications substantially degrades the performance of speech recognition systems. An example is an in-car scenario where a speech recognition system has to support the man-machine interface. Several sources of noise coming from the engine, wipers, wheels etc., interact with speech. Special challenge is given in an open window scenario, where noise of traffic, park noise, etc., has to be regarded. The main goal of this thesis is to improve the performance of a speech recognition system based on a state-of-the-art hidden Markov model (HMM) using noise reduction methods. The performance is measured with respect to word error rate and with the method of mutual information. The noise reduction methods are based on weighting rules. Least-squares weighting rules in the frequency domain have been developed to enable a continuous development based on the existing system and also to guarantee its low complexity and footprint for applications in embedded devices. The weighting rule parameters are optimized employing a multidimensional optimization task method of Monte Carlo followed by a compass search method. Root compression and cepstral smoothing methods have also been implemented to boost the recognition performance. The additional complexity and memory requirements of the proposed system are minimum. The performance of the proposed system was compared to the European Telecommunications Standards Institute (ETSI) standardized system. The proposed system outperforms the ETSI system by up to 8.6 % relative increase in word accuracy and achieves up to 35.1 % relative increase in word accuracy compared to the existing baseline system on the ETSI Aurora 3 German task. A relative increase of up to 18 % in word accuracy over the existing baseline system is also obtained from the proposed weighting rules on large vocabulary databases. An entropy-based feature vector analysis method has also been developed to assess the quality of feature vectors. The entropy estimation is based on the histogram approach. The method has the advantage to objectively asses the feature vector quality regardless of the acoustic modeling assumption used in the speech recognition system

Metalearning-based alternating minimization algorithm for nonconvex optimization

In this article, we propose a novel solution for nonconvex problems of multiple variables, especially for those typically solved by an alternating minimization (AM) strategy that splits the original optimization problem into a set of subproblems corresponding to each variable and then iteratively optimizes each subproblem using a fixed updating rule. However, due to the intrinsic nonconvexity of the original optimization problem, the optimization can be trapped into a spurious local minimum even when each subproblem can be optimally solved at each iteration. Meanwhile, learning-based approaches, such as deep unfolding algorithms, have gained popularity for nonconvex optimization; however, they are highly limited by the availability of labeled data and insufficient explainability. To tackle these issues, we propose a meta-learning based alternating minimization (MLAM) method that aims to minimize a part of the global losses over iterations instead of carrying minimization on each subproblem, and it tends to learn an adaptive strategy to replace the handcrafted counterpart resulting in advance on superior performance. The proposed MLAM maintains the original algorithmic principle, providing certain interpretability. We evaluate the proposed method on two representative problems, namely, bilinear inverse problem: matrix completion and nonlinear problem: Gaussian mixture models. The experimental results validate the proposed approach outperforms AM-based methods