An Adaptive Locally Connected Neuron Model: Focusing Neuron

This paper presents a new artificial neuron model capable of learning its receptive field in the topological domain of inputs. The model provides adaptive and differentiable local connectivity (plasticity) applicable to any domain. It requires no other tool than the backpropagation algorithm to learn its parameters which control the receptive field locations and apertures. This research explores whether this ability makes the neuron focus on informative inputs and yields any advantage over fully connected neurons. The experiments include tests of focusing neuron networks of one or two hidden layers on synthetic and well-known image recognition data sets. The results demonstrated that the focusing neurons can move their receptive fields towards more informative inputs. In the simple two-hidden layer networks, the focusing layers outperformed the dense layers in the classification of the 2D spatial data sets. Moreover, the focusing networks performed better than the dense networks even when 70$\%$ of the weights were pruned. The tests on convolutional networks revealed that using focusing layers instead of dense layers for the classification of convolutional features may work better in some data sets.Comment: 45 pages, a national patent filed, submitted to Turkish Patent Office, No: -2017/17601, Date: 09.11.201

How a student becomes a teacher: learning and forgetting through Spectral methods

In theoretical ML, the teacher-student paradigm is often employed as an effective metaphor for real-life tuition. The above scheme proves particularly relevant when the student network is overparameterized as compared to the teacher network. Under these operating conditions, it is tempting to speculate that the student ability to handle the given task could be eventually stored in a sub-portion of the whole network. This latter should be to some extent reminiscent of the frozen teacher structure, according to suitable metrics, while being approximately invariant across different architectures of the student candidate network. Unfortunately, state-of-the-art conventional learning techniques could not help in identifying the existence of such an invariant subnetwork, due to the inherent degree of non-convexity that characterizes the examined problem. In this work, we take a leap forward by proposing a radically different optimization scheme which builds on a spectral representation of the linear transfer of information between layers. The gradient is hence calculated with respect to both eigenvalues and eigenvectors with negligible increase in terms of computational and complexity load, as compared to standard training algorithms. Working in this framework, we could isolate a stable student substructure, that mirrors the true complexity of the teacher in terms of computing neurons, path distribution and topological attributes. When pruning unimportant nodes of the trained student, as follows a ranking that reflects the optimized eigenvalues, no degradation in the recorded performance is seen above a threshold that corresponds to the effective teacher size. The observed behavior can be pictured as a genuine second-order phase transition that bears universality traits.Comment: 10 pages + references + supplemental material. Poster presentation at NeurIPS 202

A study on adaptive locally connected neuron nodel

Bu çalışmada uyarlanır yerel bağlı (odaklanan) nöron modelinin bir incelemesi sunulmuştur. Öncelikle bu modelin varolan diğer nöron modelleri ile ilişkisi incelenmiştir. Daha sonra modelin ileri beslemede çalışması ve geriye yayılım ile eğitilmesi tartışılmıştır. Modelin çalışma prensipleri sentetik sınıflandırma veri kümeleri üzerinde deneylerle gösterilmiştir. Son olarak, basit ve evrişimli ağların saklı katmanlarında odaklı nöronlar kullanılması halinde tam bağlı nöronlara göre daha iyi bir performans elde edilebileceği MNIST, CIFAR10, FASHION gibi popüler imge tanıma veri kümelerinde karşılaştırmalı olarak gösterilmiştir.The manuscript presents a detailed study of adaptive local connected (focusing) neuron model. Our analysis starts with the model’s relation to other neuron models. Then we describe the feed-forward operation and its training with backpropagation gradient descent algorithm. The operation principles of the model were demonstrated with synthetically sampled data sets. Finally, the comparative experiments on popular image recognition datasets such as MNIST, CIFAR10, and FASHION show that using focusing neuron layers can improve the classification performance in some data sets.Publisher's Versio

Identification Of Streptococcus Pyogenes Using Raman Spectroscopy

Despite the attention that Raman Spectroscopy has gained recently in the area of pathogen identification, the spectra analyses techniques are not well developed. In most scenarios, they rely on expert intervention to detect and assign the peaks of the spectra to specific molecular vibration. Although some investigators have used machine-learning techniques to classify pathogens, these studies are usually limited to a specific application, and the generalization of these techniques is not clear. Also, a wide range of algorithms have been developed for classification problems, however, there is less insight to applying such methods on Raman spectra. Furthermore, analyzing the Raman spectra requires pre-processing of the raw spectra, in particular, background removing. Various techniques are developed to remove the background of the raw spectra accurately and with or without less expert intervention. Nevertheless, as the background of the spectra varies in the different media, these methods still require expert effort adding complexity and inefficiency to the identification task. This dissertation describes the development of state-of-the-art classification techniques to identify S. pyogenes from other species, including water and other confounding background pathogens. We compared these techniques in terms of their classification accuracy, sensitivity, and specificity in addition to providing a bias-variance insight in selecting the number of principal components in a principal component analysis (PCA). It was observed that Random Forest provided a better result with an accuracy of 94.11%. Next, a novel deep learning technique was developed to remove the background of the Raman spectra and then identify the pathogen. The architecture of the network was discussed and it was found that this method yields an accuracy of 100% in our test samples. This outperforms other traditional machine learning techniques as discussed. In clinical applications of Raman Spectroscopy, the samples have confounding background creates a challenging task for the removal of the spectral background and subsequent identification of the pathogen in real- time. We tested our methodology on datasets composed of confounding background such as throat swabs from patients and discussed the robustness and generalization of the developed method. It was found that the misclassification error of the test dataset was around 3.7%. Also, the realization of the trained model is discussed in detail to provide a better understating and insight into the efficacy of the deep learning architecture. This technique provides a platform for general analysis of other pathogens in confounding environments as well

Modelação e controlo de actuadores pneumáticos utilizando redes neuronais artificiais

Tese de doutoramento. Engenharia Mecânica. Faculdade de Engenharia. Universidade do Porto. 200

Modelação e controlo de actuadores pneumáticos utilizando redes neuronais artificiais

Tese de doutoramento. Engenharia Mecânica. Faculdade de Engenharia. Universidade do Porto. 200

A Neural Network Linking Process

A novel method of integrating multiple neural networks into one large network via a process referred to as a neural network linking process is proposed. Neural networks are commonly trained to solve a specific problem for an encapsulated problem domain. A single network can undertake simple classification or generalisation problems. Dividing them into sub-problems, which in turn are solved by a sub-network, can disentangle more complicated classification or generalisation problems. A controller generally combines sub-network results. A controller can be, for instance, a gating network, voting system or a mathematical combiner. In each case, every sub-network is used as a separate unit and is not interconnected to any other sub-network. However, with the linking process a novel method for linking trained sub-networks into one large network by maintaining the knowledge of each individual sub-network is introduced. Furthermore, the linked network will utilize a stimulus process in order to distinguish the type of sub-problem to be solved, by largely retaining the accuracy of the sub-network, as well as being one step closer to the biological reality