Shoplifting detection from customer behavior using deep learning

Abstract

The increase in shoplifting in the retail market causes significant stock and profit losses. Existing security methods are often costly, prone to human error, and not applicable to all product types, highlighting the need for an innovative, low-cost, and effective solution against theft. This study presents a deep learning-based system for detecting shoplifting behavior from surveillance video footage. The system integrates four components: (1) person detection to identify customers approaching shelves, (2) activity recognition to analyze movements for suspicious behavior, (3) product detection to determine which items are taken, and (4) person re-identification model which matches suspicious customers when they arrive at the checkout were developed. A Time Distributed CNN-LSTM model was developed for activity recognition; YOLOv4 was fine-tuned for person and product detection, and Siamese Networks were used for person re-identification. Training and testing were conducted using a data set collected from both an office demo setup and a real retail environment, covering five different shoplifting scenarios. The dataset collected includes 1219 videos across five scenarios. The proposed system was evaluated on a custom-collected dataset and achieved 95% overall accuracy, with component-level accuracy of 85% for activity recognition, 97% for person and product detection, and 87% for person re-identification. In this paper, the authors suggested a model which primarily focuses on recognizing shoplifting actions. The originality of this study lies in the integrated system, including 4 components; person detection, activity recognition, product detection and person re-identification which work simultaneously to provide end-to-end solutions.TÜBİTA