Enhancing Prediction Reliability Of Deep Learning By Data Confidence For Recommendation Systems: A Case Study On Named Entity Recognition

Reliability is crucial for industrial recommendation systems. Recent advancement in deep neural networks has greatly improved the performance of modern recommendation systems. However, there is a lack of research on estimating how reliable such recommendation systems are in practical scenarios. Due to the blackbox nature of the deep learning-based systems, many times additional labor has to be involved to examine the prediction accuracy manually, which is costly and time-consuming. To address the problem, we propose a novel approach to estimate the model confidence for a deep learning-based recommendation system. Our approach utilized data statistics to improve the traditional model confidence estimation and maintain the model’s high performance. We further proposed a new evaluation metric to properly compare different prediction confidence estimation approaches. Experimental results showed that the external data statistics could effectively improve the prediction reliability by increasing confidence score, which will lead to significant reduction of the time and labors on the system’s prediction result examination. Index Terms—Prediction Reliability, Recommendation Systems, Deep Learning, Data Confidence, Named Entity Recognitio