Learning how to modify training rates in scene-recognition CNNS

Abstract

Máster en Image Processing and Computer VisionIn this Master’s Thesis, we pretend to find a measure that allows modifying the value of the learning rate of each individual neuron in a convolutional neural network. Specifically, we aim to handle the effect of the phenomenon known as Catastrophic Forgetting. Starting from a network trained for a source task, this concept refers to the loss in performance that the network undergoes for the source task, when it is trained for a new target task. To this aim, we begin by adapting a neural networks visualization tool to draw conclusions about the behavior and activity of neurons. Using this information, we hypothesize that those neurons with higher activity along the data-set images may be considered useful for the source task and those with lower activity are prone to be treated as free space for the network to learn the target task. To quantitative account for this activity, we leverage on the entropy of the distribution of the neurons’ activities to design weighting functions to dynamically adapt the learning rate of each neuron according to it. In the evaluation section, we compare the results of these functions against a classical fine-tuning strategy focusing on obtaining networks whose joint performance for the source task and the target task is as close as possible to the performance obtained by two different networks fully-trained for each task separately. Obtained results suggest that all of the proposed functions perform better than the fine-tuning strategy in this scope, and some of them perform close to the fullytraining paradigm