Generative Neural Networks for Multi-task Life-Long Learning

Multi-task learning (MTL) is an established method of inducing bias in neural network learning. It has been used as a method of knowledge transfer in sequential learning tasks, and as a method of overcoming the problem of impoverished training data. A standard problem in neural networks is the establishment of the proper network structure for a given problem. This problem is compounded in the MTL case since the structure of the network can significantly impact the effectiveness of the MTL method. This problem is exacerbated further in the sequential or life-long learning case, as choosing structure a priori for an unknown number of tasks is not possible. In this paper, we explore the use of cascade correlation (CC) as a generative neural network algorithm to overcome these limitations in MTL. We show that generative neural networks using the CC algorithm can display the same MTL advantages while alleviating the need to choose the network structure a priori. We then incorporate the functional transfer from the MTL techniques with the generative structure of CC networks to build a dual network life-long learning system that does not require the network topology or the number of tasks to be predefined

Generative Neural Networks For Multi-Task Life-Long Learning

Multi-task learning (MTL) is an established method of inducing bias in neural network learning. It has been used as a method of knowledge transfer in sequential learning tasks, and as a method of overcoming the problem of impoverished training data. A standard problem in neural networks is the establishment of the proper network structure for a given problem. This problem is compounded in the MTL case since the structure of the network can significantly impact the effectiveness of the MTL method. This problem is exacerbated further in the sequential or life-long learning case, as choosing structure a priori for an unknown number of tasks is not possible. In this paper, we explore the use of cascade correlation (CC) as a generative neural network algorithm to overcome these limitations in MTL. We show that generative neural networks using the CC algorithm can display the same MTL advantages while alleviating the need to choose the network structure a priori. We then incorporate the functional transfer from the MTL techniques with the generative structure of CC networks to build a dual network life-long learning system that does not require the network topology or the number of tasks to be predefined. © The British Computer Society 2013