Mutual Information Decay Curves and Hyper-Parameter Grid Search Design for Recurrent Neural Architectures

We present an approach to design the grid searches for hyper-parameter optimization for recurrent neural architectures. The basis for this approach is the use of mutual information to analyze long distance dependencies (LDDs) within a dataset. We also report a set of experiments that demonstrate how using this approach, we obtain state-of-the-art results for DilatedRNNs across a range of benchmark datasets.Comment: Published at the 27th International Conference on Neural Information Processing, ICONIP 2020, Bangkok, Thailand, November 18-22, 2020. arXiv admin note: text overlap with arXiv:1810.0296