Recommending Tags for Images: Deep Learning Approaches for Personalized Tag Recommendation

Social media has become an integral part of numerous individuals as well as organizations, with many services being used frequently by a majority of people. Along with its widespread use, the amount of information explodes when people use these services. This demands for efficient tools as well as methods to assist data management and retrieval. Annotating resources by keywords, known as the tagging task, is a solution to improve categorizability and findability of resources. However, tagging is a human, time-consuming task, which requires the user's focus to figure out many keywords in a short moment and manually enter them into the system. To encourage users to tag their resources more correctly and frequently, tag recommendation is adopted into the social tagging systems to suggest relevant keywords for resources. In this thesis, we will address the problem of personalized tag recommendation for images and present ways to solve this problem by combining the advantages of the user relation with the images' content. In order to suggest tags for unobserved images, their visual contents are used to replace the index-based information of the image entity in the tagging relations. Because the limitation of low-level features does not show the "content" of images, we propose to utilize a deep learning based approach to learn high-level visual features concurrently with the scoring-tag estimator. For the tag predictor, a latent factor model or a multi-layer perceptron is selected to compute scores of tags by which the top selected tags are sorted in descending order. As a further development upon our findings, we examine the inside and outside context of images to enhance the accuracy of estimators. Regarding the image-inside context, we are motivated by the fact that objects, such as cars or cats are influential on the user's selection criteria. Regarding the image-outside context, the image's surrounding text contributes to the clarity of the image's content for different users. We consider these contextual features as a supporting part which is combined with the mainly visual representation to enhance the tag recommendation performance. Finally, as an additional technique, transfer learning is also adapted to support the proposed models to overcome the limitations of too small training data and boost up their performance. This thesis demonstrates the usefulness and versatility of deep learning approaches for tag recommendation and highlights the importance of the learned image's content in predicting personalized tags. Directions for future work include semantic enhancements to context-based representation and extensions of the content-aware approaches to different recommendation scenarios

Deep Learning based Recommender System: A Survey and New Perspectives

With the ever-growing volume of online information, recommender systems have been an effective strategy to overcome such information overload. The utility of recommender systems cannot be overstated, given its widespread adoption in many web applications, along with its potential impact to ameliorate many problems related to over-choice. In recent years, deep learning has garnered considerable interest in many research fields such as computer vision and natural language processing, owing not only to stellar performance but also the attractive property of learning feature representations from scratch. The influence of deep learning is also pervasive, recently demonstrating its effectiveness when applied to information retrieval and recommender systems research. Evidently, the field of deep learning in recommender system is flourishing. This article aims to provide a comprehensive review of recent research efforts on deep learning based recommender systems. More concretely, we provide and devise a taxonomy of deep learning based recommendation models, along with providing a comprehensive summary of the state-of-the-art. Finally, we expand on current trends and provide new perspectives pertaining to this new exciting development of the field.Comment: The paper has been accepted by ACM Computing Surveys. https://doi.acm.org/10.1145/328502

Modèle hybride combinant réseau de neurones convolutifs et modèle basé sur le choix pour la recommandation de sièges

Avec la vente de billets en ligne, les consommateurs souhaitant réserver un ticket pour un concert, une pièce de théâtre ou un film ont désormais la possibilité de choisir leur emplacement. Ce choix influence l’expérience vécue : différents facteurs sont à considérer, et chaque client fait son propre raisonnement (plus ou moins consciemment) pour prendre cette décision. Par exemple, dans un cinéma, certaines personnes vont privilégier les sièges au centre pour avoir la meilleure vision possible de l’écran, tandis que d’autres pourront préférer les sièges latéraux pour être moins dérangés par la présence d’autrui, en particulier si beaucoup de sièges au centre sont déjà réservés. Cet exemple illustre l’hétérogénéité de raisonnement d’un consommateur dans cette situation, et met en valeur deux catégories de facteurs influant sur la prise de décision : la position dans la salle, et la proximité aux autres. La réservation en ligne a ainsi permis de collecter ces choix dans des bases de données, et pour l’industrie culturelle (dans notre exemple le gérant de cinéma), cette information peut être cruciale. D’abord, connaître les sièges les plus attractifs à un instant donné peut permettre de modifier la tarification et ainsi augmenter l’affluence dans les salles et donc les recettes. De plus, si cette connaissance se fait spécifiquement pour chaque utilisateur ayant déjà effectué des réservations par le passé, cela peut également permettre d’améliorer les stratégies marketing par la mise en place d’un système de recommandation personnalisé de sièges. Un premier objectif du mémoire est la revue de méthodes permettant l’estimation de l’attractivité d’un siège dans une salle partiellement remplie. Deux stratégies sont possibles : la première consiste à traiter chaque client individuellement afin d’assurer une modélisation personnelle de la prise de décision, mais qui est limité par la quantité de données disponible par clients. L’autre stratégie consiste à regrouper l’ensemble des données pour pouvoir appliquer des modèles avec plus de capacité comme de l’apprentissage profond, mais qui perd l’information du comportement individuel. Une hypothèse de ce mémoire est que malgré une performance plus faible pour la deuxième stratégie, cette dernière apporte de l’information utile, et une combinaison des deux permet d’améliorer la performance globale et de pallier au problème de la stratégie individualisée du possible manque de données.----------ABSTRACT: With online ticket sales, consumers wishing to book a ticket for a concert or a movie now have the opportunity to choose their location. This choice influences the lived experience: different factors have to be considered, and each client makes his own reasoning (more or less consciously) to make this decision. For example, in a movie theatre, some people may prefer centre seats to get the best possible view of the screen, while others may prefer side seats to be less disturbed by the presence of others, especially if many centre seats are already reserved. This example illustrates the heterogeneity of reasoning of a consumer in this situation, and highlights two categories of factors influencing decision making: position in the room, and proximity to others. Online booking has thus made it possible to collect these choices in databases, and for the cultural industry, this information can be crucial. Firstly, knowing the most attractive seats at a given time can help to modify the pricing and thus increase attendance in halls and thus revenues. Moreover, if this knowledge is done specifically for each user who has made reservations in the past, it can also help improve marketing strategies by implementing a personalized seat recommendation system. A first objective here is the review of methods for estimating the attractiveness of a seat in a partially-filled room. Two strategies are possible: the first one is to treat each client individually to ensure personal modeling of decision making, but this is limited by the amount of data available per client. The other strategy is to aggregate the data to be able to apply models with more capacity such as deep learning, but lose the information about individual behaviour. One hypothesis of this paper is that despite weaker performance for the second strategy, the latter provides useful information, and a combination of the two can improve overall performance and overcome the problem of the individualized strategy of the possible lack of data

Quantum neural networks

Quantum computing is one of the most exciting research areas of the last decades. At the same time, methods of machine learning have started to dominate science, industry and our everyday life. In this thesis we combine these two essential research topics of the 21st century and introduce dissipative quantum neural networks (DQNNs), which are designed for fully quantum learning tasks, are capable of universal quantum computation and have low memory requirements while training. We start the discussion of this interdisciplinary topic by introducing artificial neural networks, which are a very common tool in classical machine learning. Next, we give an overview on quantum information. Here we focus on quantum algorithms and circuits, which are used to implement quantum neural networks. Moreover, we explain the opportunities and challenges arising with today's quantum computers. The discussion of the architecture and training algorithm of the DQNNs forms the core of this work. These networks are optimised with training data pairs in form of input and desired output states and therefore can be used for characterising unknown or untrusted quantum devices. We not only demonstrate the generalisation behaviour of these quantum neural networks using classical simulations, but also implement them successfully on actual quantum computers. To understand the ultimate limits for such quantum machine learning methods, we discuss the quantum no free lunch theorem, which describes a bound on the probability that a quantum device, which can be modelled as a unitary process and is optimised with quantum examples, gives an incorrect output for a random input. This gives us a tool to review the learning behaviour of quantum neural networks in general and the DQNNs in particular. Moreover we expand the area of applications of DQNNs in two directions. In the first case, we include additional information beyond just the training data pairs: since quantum devices are always structured, the resulting data is always structured as well. We modify the DQNN's training algorithm such that knowledge about the graph-structure of the training data pairs is included in the training process and show that this can lead to better generalisation behaviour. Both the original DQNN and the DQNN including graph structure are trained with data pairs in order to characterise an underlying relation. However, in the second extension of the algorithm we aim to learn characteristics of a set of quantum states in order to extend it to quantum states which have similar properties. Therefore we build a generative adversarial model where two DQNNs, called the generator and discriminator, are trained in a competitive way. Overall, we observe that DQNNs can not only be trained efficiently but also, similar to their classical counterparts, modified to suit different applications.Quantencomputer bilden eines der spannendsten Forschungsgebiete der letzten Jahrzehnte. Zur gleichen Zeit haben Methoden des maschinellen Lernens begonnen die Wissenschaft, Industrie und unseren Alltag zu dominieren. In dieser Arbeit kombinieren wir diese beiden wichtigen Forschungsthemen des 21. Jahrhunderts und stellen dissipative quantenneuronale Netze (DQNNs) vor, die für Quantenlernaufgaben konzipiert sind, universelle Quantenberechnungen durchführen können und wenig Speicherbedarf beim Training benötigen. Wir beginnen die Diskussion dieses interdisziplinären Themas mit der Einführung künstlicher neuronaler Netze, die beim klassischen maschinellen Lernen weit verbreitet sind. Dann geben wir einen Überblick über die Quanteninformationstheorie. Hier fokussieren wir uns auf die zur Implementierung von quantenneuronalen Netzen nötigen Quantenalgorithmen und -schaltungen. Außerdem erläutern wir die Chancen und Herausforderungen der heutigen Quantencomputer. Die Diskussion der Architektur und des Trainingsalgorithmus der DQNNs bildet den Mittelpunkt dieser Arbeit. Diese Netzwerke werden mit Trainingsdatenpaaren in Form von Eingangs- und gewünschten Ausgangszuständen optimiert und können daher zur Charakterisierung unbekannter oder nicht vertrauenswürdiger Quantenbauelemente verwendet werden. Wir demonstrieren nicht nur das Generalisierungsverhalten dieser Netze anhand klassischer Simulationen, sondern konstruieren auch eine erfolgreiche Implementierung für Quantencomputer. Um die ultimativen Grenzen solcher Methoden zum maschinellen Lernen von Quantendaten zu verstehen, führen wir das quantum no free lunch-Theorem ein, welches eine Begrenzung für die Wahrscheinlichkeit beschreibt, dass ein als unitärer Prozess modellierbares und mit Quantendaten optimiertes Quantenbauelement eine falsche Ausgabe für eine zufällige Eingabe herausgibt. Das Theorem gibt uns ein Werkzeug, um das Lernverhalten von quantenneuronalen Netzwerken im Allgemeinen und der DQNNs im Besonderen zu überprüfen. Darüber hinaus erweitern wir den Anwendungsbereich von DQNNs auf zwei Weisen. Im ersten Fall beziehen wir Informationen zusätzlich zu den Trainingsdaten mit ein: Da Quantenbauelemente immer eine gewisse Struktur haben, sind auch die resultierenden Daten strukturiert. Wir modifizieren den Trainingsalgorithmus der DQNNs so, dass Kenntnisse über die Struktur genutzt werden können und zeigen, dass dies zu einem besseren Trainingsergebnis führen kann. Sowohl das ursprüngliche DQNN als auch das Graphen-DQNN wird mit Datenpaaren trainiert, um eine zugrunde liegende Relation zu charakterisieren. Als zweite Erweiterung wollen wir jedoch die Eigenschaften einer Menge einzelner Quantenzustände untersuchen, um sie mit Quantenzuständen ähnlicher Eigenschaften zu erweitern. Daher konstruieren wir ein Modell, bei dem zwei DQNNs, Generator und Diskriminator genannt, kompetitiv trainiert werden. Zusammenfassend stellen wir fest, dass DQNNs nicht nur effizient trainiert, sondern auch, ähnlich wie ihre klassischen Gegenstücke, an unterschiedliche Anwendungen angepasst werden können

Vision transformers with Inductive Bias introduced through self-attention regularization

In recent years, the Transformer achieved remarkable results in computer vision related tasks, matching, or even surpassing those of convolutional neural networks (CNN). However, unlike CNNs, those vision transformers lack strong inductive biases and, to achieve state-of-the-art results, rely on large architectures and extensive pre-training on tens of millions of images. Introducing the appropriate inductive biases to vision transformers can lead to better convergence and generalization on settings with fewer training data. This work presents a novel way to introduce inductive biases to vision transformers: self-attention regularization. Two different methods of self-attention regularization were devised. Furthermore, this work proposes ARViT, a novel vision transformer architecture, where both self-attention regularization methods are deployed. The experimental results demonstrated that self-attention regularization leads to better convergence and generalization, especially on models pre-trained on mid-size datasets.創価大