A deep learning pipeline for product recognition on store shelves

Recognition of grocery products in store shelves poses peculiar challenges. Firstly, the task mandates the recognition of an extremely high number of different items, in the order of several thousands for medium-small shops, with many of them featuring small inter and intra class variability. Then, available product databases usually include just one or a few studio-quality images per product (referred to herein as reference images), whilst at test time recognition is performed on pictures displaying a portion of a shelf containing several products and taken in the store by cheap cameras (referred to as query images). Moreover, as the items on sale in a store as well as their appearance change frequently over time, a practical recognition system should handle seamlessly new products/packages. Inspired by recent advances in object detection and image retrieval, we propose to leverage on state of the art object detectors based on deep learning to obtain an initial productagnostic item detection. Then, we pursue product recognition through a similarity search between global descriptors computed on reference and cropped query images. To maximize performance, we learn an ad-hoc global descriptor by a CNN trained on reference images based on an image embedding loss. Our system is computationally expensive at training time but can perform recognition rapidly and accurately at test time

Product recognition in store shelves as a sub-graph isomorphism problem

The arrangement of products in store shelves is carefully planned to maximize sales and keep customers happy. However, verifying compliance of real shelves to the ideal layout is a costly task routinely performed by the store personnel. In this paper, we propose a computer vision pipeline to recognize products on shelves and verify compliance to the planned layout. We deploy local invariant features together with a novel formulation of the product recognition problem as a sub-graph isomorphism between the items appearing in the given image and the ideal layout. This allows for auto-localizing the given image within the aisle or store and improving recognition dramatically.Comment: Slightly extended version of the paper accepted at ICIAP 2017. More information @project_page --> http://vision.disi.unibo.it/index.php?option=com_content&view=article&id=111&catid=7

Concept-based Anomaly Detection in Retail Stores for Automatic Correction using Mobile Robots

Tracking of inventory and rearrangement of misplaced items are some of the most labor-intensive tasks in a retail environment. While there have been attempts at using vision-based techniques for these tasks, they mostly use planogram compliance for detection of any anomalies, a technique that has been found lacking in robustness and scalability. Moreover, existing systems rely on human intervention to perform corrective actions after detection. In this paper, we present Co-AD, a Concept-based Anomaly Detection approach using a Vision Transformer (ViT) that is able to flag misplaced objects without using a prior knowledge base such as a planogram. It uses an auto-encoder architecture followed by outlier detection in the latent space. Co-AD has a peak success rate of 89.90% on anomaly detection image sets of retail objects drawn from the RP2K dataset, compared to 80.81% on the best-performing baseline of a standard ViT auto-encoder. To demonstrate its utility, we describe a robotic mobile manipulation pipeline to autonomously correct the anomalies flagged by Co-AD. This work is ultimately aimed towards developing autonomous mobile robot solutions that reduce the need for human intervention in retail store management.Comment: 8 pages, 9 figures, 2 tables, IEEE Transactions on Systems, Man and Cybernetic

Retail Shelf Analytics Through Image Processing and Deep Learning

The present thesis promotes an innovative approach based on modern deep learning and image processing techniques for retail shelf analytics within an actual business context. To achieve this goal, the research focused on recent developments in computer vision while maintaining a business-oriented approach. The project involved the full-stack software development of a product to analyze structured and unstructured data and provide business intelligence services for retail systems

A Review of Recent Advances and Challenges in Grocery Label Detection and Recognition

When compared with traditional local shops where the customer has a personalised service, in large retail departments, the client has to make his purchase decisions independently, mostly supported by the information available in the package. Additionally, people are becoming more aware of the importance of the food ingredients and demanding about the type of products they buy and the information provided in the package, despite it often being hard to interpret. Big shops such as supermarkets have also introduced important challenges for the retailer due to the large number of different products in the store, heterogeneous affluence and the daily needs of item repositioning. In this scenario, the automatic detection and recognition of products on the shelves or off the shelves has gained increased interest as the application of these technologies may improve the shopping experience through self-assisted shopping apps and autonomous shopping, or even benefit stock management with real-time inventory, automatic shelf monitoring and product tracking. These solutions can also have an important impact on customers with visual impairments. Despite recent developments in computer vision, automatic grocery product recognition is still very challenging, with most works focusing on the detection or recognition of a small number of products, often under controlled conditions. This paper discusses the challenges related to this problem and presents a review of proposed methods for retail product label processing, with a special focus on assisted analysis for customer support, including for the visually impaired. Moreover, it details the public datasets used in this topic and identifies their limitations, and discusses future research directions of related fields.info:eu-repo/semantics/publishedVersio

Computer Vision and Deep Learning for retail store management

The management of a supermarket or retail store is a quite complex process that requires the coordinated execution of many different tasks (\eg, shelves management, inventory, surveillance, customer support\dots). Thanks to recent advancements of technology, many of those repetitive tasks can be completely or partially automated. One key technology requirement is the ability to understand a scene based only on information acquired by a camera, for this reason, we will focus on computer vision techniques to solve management problems inside a grocery retail store. We will address two main problems: (a) how to detect and recognize automatically products exposed on store shelves and (b) how to obtain a reliable 3D reconstruction of an environment using only information coming from a camera. We will tackle (a) both in a constrained version where the objective is to verify the compliance of observed items to a planned disposition, as well as an unconstrained one where no assumption on the observed scenes are considered. As for (b), a good solution represents one of the first crucial steps for the development and deployment of low-cost autonomous agents able to safely navigate inside the store either to carry out management jobs or to help customers (\eg, autonomous cart or shopping assistant). We believe that algorithms for depth prediction from stereo or mono camera are good candidates for the solution of this problem. The current state of the art algorithms, however, rely heavily on machine learning and can be hardly applied in the retail environment due to problems arising from the domain shift between data used to train them (usually synthetic images) and the deployment scenario (real indoor images). We will introduce techniques to adapt those algorithms to unseen environments without the need of costly ground truth data and in real time

Tecnología para Tiendas Inteligentes

Trabajo de Fin de Grado en Doble Grado en Ingeniería Informática y Matemáticas, Facultad de Informática UCM, Departamento de Ingeniería del Software e Inteligencia Artificial, Curso 2020/2021Smart stores technologies exemplify how Artificial Intelligence and Internet of Things can effectively join forces to shape the future of retailing. With an increasing number of companies proposing and implementing their own smart store concepts, such as Amazon Go or Tao Cafe, a new field is clearly emerging. Since the technologies used to build their infrastructure offer significant competitive advantages, companies are not publicly sharing their own designs. For this reason, this work presents a new smart store model named Mercury, which aims to take the edge off of the lack of public and accessible information and research documents in this field. We do not only introduce a comprehensive smart store model, but also work-through a feasible detailed implementation so that anyone can build their own system upon it.Las tecnologías utilizadas en las tiendas inteligentes ejemplifican cómo la Inteligencia Artificial y el Internet de las Cosas pueden unir, de manera efectiva, fuerzas para transformar el futuro de la venta al por menor. Con un creciente número de empresas proponiendo e implementando sus propios conceptos de tiendas inteligentes, como Amazon Go o Tao Cafe, un nuevo campo está claramente emergiendo. Debido a que las tecnologías utilizadas para construir sus infraestructuras ofrecen una importante ventaja competitiva, las empresas no están compartiendo públicamente sus diseños. Por esta razón, este trabajo presenta un nuevo modelo de tienda inteligente llamado Mercury, que tiene como objetivo mitigar la falta de información pública y accesible en este campo. No solo introduciremos un modelo general y completo de tienda inteligente, sino que también proponemos una implementación detallada y concreta para que cualquier persona pueda construir su propia tienda inteligente siguiendo nuestro modelo.Depto. de Ingeniería de Software e Inteligencia Artificial (ISIA)Fac. de InformáticaTRUEunpu