Impact of mislabelling on deep learning methods and strategies for improvement

Abstract

This contribution revolves around classifying football player actions with 1-dimensional convolutional neural networks (CNNs) based on 6-channel inertial motion unit (IMU) data arising from tracking devices worn by the players. Our training and test data consist of eight games, where humans labelled ball actions by inspecting video records. Unfortunately, these labels are far from perfect due to various reasons (e.g., sloppiness, not all players respectively ball actions visible in the record, ambiguity what a ball action is, etc.). Such mislabelled data provide challenges on several levels. First, performance evaluation with poorly annotated data can be strongly misleading, indicating inferior performance than what is truly achieved. Second, the question is what amount of mislabelled data deep artificial neural networks can tolerate before they break down. We try to shed some light on the magnitude of these effects by simulation studies on the football data, as well as some standard machine learning datasets such as MNIST (numbers) and Fashion-MNIST (clothes). Third, we present some efficient strategies to overcome the issue with imperfect labels and aim to provide some guidelines how to efficiently invest effort in labelling data