Comparing Neural and Attractiveness-based Visual Features for Artwork Recommendation

Advances in image processing and computer vision in the latest years have brought about the use of visual features in artwork recommendation. Recent works have shown that visual features obtained from pre-trained deep neural networks (DNNs) perform very well for recommending digital art. Other recent works have shown that explicit visual features (EVF) based on attractiveness can perform well in preference prediction tasks, but no previous work has compared DNN features versus specific attractiveness-based visual features (e.g. brightness, texture) in terms of recommendation performance. In this work, we study and compare the performance of DNN and EVF features for the purpose of physical artwork recommendation using transactional data from UGallery, an online store of physical paintings. In addition, we perform an exploratory analysis to understand if DNN embedded features have some relation with certain EVF. Our results show that DNN features outperform EVF, that certain EVF features are more suited for physical artwork recommendation and, finally, we show evidence that certain neurons in the DNN might be partially encoding visual features such as brightness, providing an opportunity for explaining recommendations based on visual neural models.Comment: DLRS 2017 workshop, co-located at RecSys 201

Presentation Bias in movie recommendation algorithms

Dissertation presented as the partial requirement for obtaining a Master's degree in Statistics and Information Management, specialization Information Analysis and ManagementThe emergence of video on demand (VOD) has transformed the way the content finds its audience. Several improvements have been made on algorithms to provide better movie recommendations to individuals. Given the huge variety of elements that characterize a film (such as casting, genre, soundtrack, amongst others artistic and technical aspects) and that characterize individuals, most of the improvements relied on accomplishing those characteristics to do a better job regarding matching potential clients to each product. However, little attention has been given to evaluate how the algorithms’ result selection are affected by presentation bias. Understanding bias is key to choosing which algorithms will be used by the companies. The existence of a system with presentation bias and feedback loop is already a problem stated by Netflix. In this sense, this research will fill that gap providing a comparative analysis of the bias of the major movie recommendation algorithms

Enhancing [Spatial] Creativity – Enhancing creativity of architects by applying unconventional virtual environments (UVEs)

Potentials of virtual environment for enhancing creativity of architects have shaped this research. There is no singular definition of creativity. In fact, there are more than 100 different definitions for creativity according to different contexts and disciplines. Nevertheless, it is possible to confine the boundaries of definitions and address creativity within a confined framework. The first practical step was thus to perform an in-depth literature survey to define a boundary condition for the widespread topic of “creativity” and identify vital research questions pertaining to creativity. In this regard, personality and behavior of creative people; mood, state, temper, intelligence vs. creativity, motivation and so forth were ignored. Instead, cognitive aspects of creativity such as thinking patterns, conceptual blending, idea expansion and tolerance of ambiguity have been focused upon.The second step was to test whether starting a design procedure with a 2 or a 3-dimensional mode of thinking has any correlation with creativity. An experiment pertaining to this test was designed in which participants were asked to perform the same design task once with 2D tools and environments (e.g. traditional pen and paper) and the next time by applying 3D tools and environments (e.g. 3D software). A jury of experts in the field of design subjectively compared the results and arrived at a conclusion that participates generated more creative ideas by implementing 3D environments/tools. The third step, involved the introduction of unconventional virtual environments (UVEs), which subsequently lead to the creation of a hypothesis. This hypothesis tries to connect navigation in UVEs with the enhancement of creativity.  Characteristics of UVEs and theoretical arguments around the hypothesis were also discussed. The fourth step, involved a discussion on two effective parameters of creativity: 1- Tolerance of ambiguity2- Conceptual blendingAttempts to verify these parameters, lead to the formulation of two separate experiments. The conclusions of these experiments were as follows:1-  Tolerance of ambiguity has a direct relationship with creativity. Architects deal with multiple parameters during a typical design process. The ability to meaningfully process the relationships between such a multitude of parameters has also trained architects to possess a higher level of tolerance of ambiguity aiding them in postponing making hasty judgements.2-  By combining different ideas, one can formulate newer and much novel ideas. Ideas tend to become creative when more remote ideas are combined and synthesized. In order to prevent repetitive and self-similar ideas, the unconscious mind needs to be fed with more ideas, which the mind has not been exposed to. These new ideas/experiences can be generated by exposing one’s self to UVEs, since UVEs can expose one to situations and experiences which the brain cannot experience in the physical world.The fifth step, aims at understanding how the brain perceives different environments. Three different environments were chosen for an experimental study pertaining to the same: 1- Abstract environment 2- Semi-designed environment and 3- Fully designed environment. Participants were asked to provide a feedback by answering a questionnaire after navigating each of these environments. Simultaneously, their brain activity patterns were recorded via a professional neuropsychology apparatus. After analyzing the brain activities, in conjunction with the questionnaire, it became clear that perception of an abstract environment is completely different from the perception of a Semi-designed or Fully designed environment.  This experiment consolidated the hypothesis that UVEs as abstract designed environments activate parts of the brain that are correlate with creativity.The sixth step, involved putting the hypothesis to a final test. A conclusive experiment was thus designed. The experiment won the Visionair fund of FP7 and was conducted in Italy. A UVE was designed using the software 3D max and was converted to 3D stereoscopic mode using a specific software: GIOVE; developed in ITIA-CNR, Italy. Using a 3D goggle and 6 axis mouse, a group of 20 participants were asked to navigate the UVE. They were asked to provide a written feedback pertaining to their feelings, expectations, strategy of navigation and in general, their experience. Their responses were collected and analyzed.The final step, involved answering the formulated research questions and discussing the final results

Enhancing [Spatial] Creativity: Enhancing creativity of architects by applying unconventional virtual environments

Potentials of a virtual environment for enhancing the creativity of architects have shaped this research. There is no singular definition of creativity. In fact, there are more than 100 different definitions of creativity according to different contexts and disciplines. Nevertheless, it is possible to confine the boundaries of definitions and address creativity within a confined framework.  The first practical step was thus to perform an in-depth literature survey to define a boundary condition for the widespread topic of “creativity†and identify vital research questions pertaining to creativity. In this regard, personality and behaviour of creative people; mood, state, temper, intelligence vs. creativity, motivation and so forth were ignored. Instead, cognitive aspects of creativity such as thinking patterns, conceptual blending, idea expansion and tolerance of ambiguity have been focused upon. The second step was to test whether starting a design procedure with a 2 or a 3-dimensional mode of thinking has any correlation with creativity. An experiment pertaining to this test was designed in which participants were asked to perform the same design task once with 2D tools and environments (e.g. traditional pen and paper) and the next time by applying 3D tools and environments (e.g. 3D software). A jury of experts in the field of design subjectively compared the results and arrived at a conclusion that participates generated more creative ideas by implementing 3D environments/tools.  The third step involved the introduction of unconventional virtual environments (UVEs), which subsequently lead to the creation of a hypothesis. This hypothesis tries to connect navigation in UVEs with the enhancement of creativity.  Characteristics of UVEs and theoretical arguments around the hypothesis were also discussed.  The fourth step involved a discussion on two effective parameters of creativity: Tolerance of ambiguity Conceptual blending Attempts to verify these parameters, lead to the formulation of two separate experiments. The conclusions of these experiments were as follows: Tolerance of ambiguity has a direct relationship with creativity. Architects deal with multiple parameters during a typical design process. The ability to meaningfully process the relationships between such a multitude of parameters has also trained architects to possess a higher level of tolerance of ambiguity aiding them in postponing making hasty judgements. By combining different ideas, one can formulate newer and much novel ideas. Ideas tend to become creative when more remote ideas are combined and synthesised. In order to prevent repetitive and self-similar ideas, the unconscious mind needs to be fed with more ideas, which the mind has not been exposed to. These new ideas/experiences can be generated by exposing one’s self to UVEs since UVEs can expose one to situations and experiences which the brain cannot experience in the physical world. The fifth step aims at understanding how the brain perceives different environments. Three different environments were chosen for an experimental study pertaining to the same: 1- Abstract environment 2- Semi-designed environment and 3- Fully designed environment. Participants were asked to provide a feedback by answering a questionnaire after navigating each of these environments. Simultaneously, their brain activity patterns were recorded via a professional neuropsychology apparatus. After analysing the brain activities, in conjunction with the questionnaire, it became clear that perception of an abstract environment is completely different from the perception of a Semi-designed or Fully designed environment.  This experiment consolidated the hypothesis that UVEs as abstract designed environments activate parts of the brain that are correlated with creativity. The sixth step involved putting the hypothesis to a final test. A conclusive experiment was thus designed. The experiment won the Visionair fund of FP7 and was conducted in Italy. A UVE was designed using the software 3D max and was converted to the 3D stereoscopic mode using a specific software: GIOVE; developed in ITIA-CNR, Italy. Using a 3D goggle and 6 axis mouse, a group of 20 participants were asked to navigate the UVE. They were asked to provide a written feedback pertaining to their feelings, expectations, the strategy of navigation and in general, their experience. Their responses were collected and analysed. The final step involved answering the formulated research questions and discussing the final results

KEER2022

Avanttítol: KEER2022. DiversitiesDescripció del recurs: 25 juliol 202

Building a semantic search engine with games and crowdsourcing

Semantic search engines aim at improving conventional search with semantic information, or meta-data, on the data searched for and/or on the searchers. So far, approaches to semantic search exploit characteristics of the searchers like age, education, or spoken language for selecting and/or ranking search results. Such data allow to build up a semantic search engine as an extension of a conventional search engine. The crawlers of well established search engines like Google, Yahoo! or Bing can index documents but, so far, their capabilities to recognize the intentions of searchers are still rather limited. Indeed, taking into account characteristics of the searchers considerably extend both, the quantity of data to analyse and the dimensionality of the search problem. Well established search engines therefore still focus on general search, that is, "search for all", not on specialized search, that is, "search for a few". This thesis reports on techniques that have been adapted or conceived, deployed, and tested for building a semantic search engine for the very specific context of artworks. In contrast to, for example, the interpretation of X-ray images, the interpretation of artworks is far from being fully automatable. Therefore artwork interpretation has been based on Human Computation, that is, a software-based gathering of contributions by many humans. The approach reported about in this thesis first relies on so called Games With A Purpose, or GWAPs, for this gathering: Casual games provide an incentive for a potentially unlimited community of humans to contribute with their appreciations of artworks. Designing convenient incentives is less trivial than it might seem at first. An ecosystem of games is needed so as to collect the meta-data on artworks intended for. One game generates the data that can serve as input of another game. This results in semantically rich meta-data that can be used for building up a successful semantic search engine. Thus, a first part of this thesis reports on a "game ecosystem" specifically designed from one known game and including several novel games belonging to the following game classes: (1) Description Games for collecting obvious and trivial meta-data, basically the well-known ESP (for extra-sensorial perception) game of Luis von Ahn, (2) the Dissemination Game Eligo generating translations, (3) the Diversification Game Karido aiming at sharpening differences between the objects, that is, the artworks, interpreted and (3) the Integration Games Combino, Sentiment and TagATag that generate structured meta-data. Secondly, the approach to building a semantic search engine reported about in this thesis relies on Higher-Order Singular Value Decomposition (SVD). More precisely, the data and meta-data on artworks gathered with the afore mentioned GWAPs are collected in a tensor, that is a mathematical structure generalising matrices to more than only two dimensions, columns and rows. The dimensions considered are the artwork descriptions, the players, and the artwork themselves. A Higher-Order SVD of this tensor is first used for noise reduction in This thesis reports also on deploying a Higher-Order LSA. The parallel Higher-Order SVD algorithm applied for the Higher-Order LSA and its implementation has been validated on an application related to, but independent from, the semantic search engine for artworks striven for: image compression. This thesis reports on the surprisingly good image compression which can be achieved with Higher-Order SVD. While compression methods based on matrix SVD for each color, the approach reported about in this thesis relies on one single (higher-order) SVD of the whole tensor. This results in both, better quality of the compressed image and in a significant reduction of the memory space needed. Higher-Order SVD is extremely time-consuming what calls for parallel computation. Thus, a step towards automatizing the construction of a semantic search engine for artworks was parallelizing the higher-order SVD method used and running the resulting parallel algorithm on a super-computer. This thesis reports on using Hestenes’ method and R-SVD for parallelising the higher-order SVD. This method is an unconventional choice which is explained and motivated. As of the super-computer needed, this thesis reports on turning the web browsers of the players or searchers into a distributed parallel computer. This is done by a novel specific system and a novel implementation of the MapReduce data framework to data parallelism. Harnessing the web browsers of the players or searchers saves computational power on the server-side. It also scales extremely well with the number of players or searchers because both, playing with and searching for artworks, require human reflection and therefore results in idle local processors that can be brought together into a distributed super-computer.Semantische Suchmaschinen dienen der Verbesserung konventioneller Suche mit semantischen Informationen, oder Metadaten, zu Daten, nach denen gesucht wird, oder zu den Suchenden. Bisher nutzt Semantische Suche Charakteristika von Suchenden wie Alter, Bildung oder gesprochene Sprache für die Auswahl und/oder das Ranking von Suchergebnissen. Solche Daten erlauben den Aufbau einer Semantischen Suchmaschine als Erweiterung einer konventionellen Suchmaschine. Die Crawler der fest etablierten Suchmaschinen wie Google, Yahoo! oder Bing können Dokumente indizieren, bisher sind die Fähigkeiten eher beschränkt, die Absichten von Suchenden zu erkennen. Tatsächlich erweitert die Berücksichtigung von Charakteristika von Suchenden beträchtlich beides, die Menge an zu analysierenden Daten und die Dimensionalität des Such-Problems. Fest etablierte Suchmaschinen fokussieren deswegen stark auf allgemeine Suche, also "Suche für alle", nicht auf spezialisierte Suche, also "Suche für wenige". Diese Arbeit berichtet von Techniken, die adaptiert oder konzipiert, eingesetzt und getestet wurden, um eine semantische Suchmaschine für den sehr speziellen Kontext von Kunstwerken aufzubauen. Im Gegensatz beispielsweise zur Interpretation von Röntgenbildern ist die Interpretation von Kunstwerken weit weg davon gänzlich automatisiert werden zu können. Deswegen basiert die Interpretation von Kunstwerken auf menschlichen Berechnungen, also Software-basiertes Sammeln von menschlichen Beiträgen. Der Ansatz, über den in dieser Arbeit berichtet wird, beruht auf sogenannten "Games With a Purpose" oder GWAPs die folgendes sammeln: Zwanglose Spiele bieten einen Anreiz für eine potenziell unbeschränkte Gemeinde von Menschen, mit Ihrer Wertschätzung von Kunstwerken beizutragen. Geeignete Anreize zu entwerfen in weniger trivial als es zuerst scheinen mag. Ein Ökosystem von Spielen wird benötigt, um Metadaten gedacht für Kunstwerke zu sammeln. Ein Spiel erzeugt Daten, die als Eingabe für ein anderes Spiel dienen können. Dies resultiert in semantisch reichhaltigen Metadaten, die verwendet werden können, um eine erfolgreiche Semantische Suchmaschine aufzubauen. Deswegen berichtet der erste Teil dieser Arbeit von einem "Spiel-Ökosystem", entwickelt auf Basis eines bekannten Spiels und verschiedenen neuartigen Spielen, die zu verschiedenen Spiel-Klassen gehören. (1) Beschreibungs-Spiele zum Sammeln offensichtlicher und trivialer Metadaten, vor allem dem gut bekannten ESP-Spiel (Extra Sensorische Wahrnehmung) von Luis von Ahn, (2) dem Verbreitungs-Spiel Eligo zur Erzeugung von Übersetzungen, (3) dem Diversifikations-Spiel Karido, das Unterschiede zwischen Objekten, also interpretierten Kunstwerken, schärft und (3) Integrations-Spiele Combino, Sentiment und Tag A Tag, die strukturierte Metadaten erzeugen. Zweitens beruht der Ansatz zum Aufbau einer semantischen Suchmaschine, wie in dieser Arbeit berichtet, auf Singulärwertzerlegung (SVD) höherer Ordnung. Präziser werden die Daten und Metadaten über Kunstwerk gesammelt mit den vorher genannten GWAPs in einem Tensor gesammelt, einer mathematischen Struktur zur Generalisierung von Matrizen zu mehr als zwei Dimensionen, Spalten und Zeilen. Die betrachteten Dimensionen sind die Beschreibungen der Kunstwerke, die Spieler, und die Kunstwerke selbst. Eine Singulärwertzerlegung höherer Ordnung dieses Tensors wird zuerst zur Rauschreduktion verwendet nach der Methode der sogenannten Latenten Semantischen Analyse (LSA). Diese Arbeit berichtet auch über die Anwendung einer LSA höherer Ordnung. Der parallele Algorithmus für Singulärwertzerlegungen höherer Ordnung, der für LSA höherer Ordnung verwendet wird, und seine Implementierung wurden validiert an einer verwandten aber von der semantischen Suche unabhängig angestrebten Anwendung: Bildkompression. Diese Arbeit berichtet von überraschend guter Kompression, die mit Singulärwertzerlegung höherer Ordnung erzielt werden kann. Neben Matrix-SVD-basierten Kompressionsverfahren für jede Farbe, beruht der Ansatz wie in dieser Arbeit berichtet auf einer einzigen SVD (höherer Ordnung) auf dem gesamten Tensor. Dies resultiert in beidem, besserer Qualität von komprimierten Bildern und einer signifikant geringeren des benötigten Speicherplatzes. Singulärwertzerlegung höherer Ordnung ist extrem zeitaufwändig, was parallele Berechnung verlangt. Deswegen war ein Schritt in Richtung Aufbau einer semantischen Suchmaschine für Kunstwerke eine Parallelisierung der verwendeten SVD höherer Ordnung auf einem Super-Computer. Diese Arbeit berichtet vom Einsatz der Hestenes’-Methode und R-SVD zur Parallelisierung der SVD höherer Ordnung. Diese Methode ist eine unkonventionell Wahl, die erklärt und motiviert wird. Ab nun wird ein Super-Computer benötigt. Diese Arbeit berichtet über die Wandlung der Webbrowser von Spielern oder Suchenden in einen verteilten Super-Computer. Dies leistet ein neuartiges spezielles System und eine neuartige Implementierung des MapReduce Daten-Frameworks für Datenparallelismus. Das Einspannen der Webbrowser von Spielern und Suchenden spart server-seitige Berechnungskraft. Ebenso skaliert die Berechnungskraft so extrem gut mit der Spieleranzahl oder Suchenden, denn beides, Spiel mit oder Suche nach Kunstwerken, benötigt menschliche Reflektion, was deswegen zu ungenutzten lokalen Prozessoren führt, die zu einem verteilten Super-Computer zusammengeschlossen werden können

Development of Context-Aware Recommenders of Sequences of Touristic Activities

En els últims anys, els sistemes de recomanació s'han fet omnipresents a la xarxa. Molts serveis web, inclosa la transmissió de pel·lícules, la cerca web i el comerç electrònic, utilitzen sistemes de recomanació per facilitar la presa de decisions. El turisme és una indústria molt representada a la xarxa. Hi ha diversos serveis web (e.g. TripAdvisor, Yelp) que es beneficien de la integració de sistemes recomanadors per ajudar els turistes a explorar destinacions turístiques. Això ha augmentat la investigació centrada en la millora dels recomanadors turístics per resoldre els principals problemes als quals s'enfronten. Aquesta tesi proposa nous algorismes per a sistemes recomanadors turístics que aprenen les preferències dels turistes a partir dels seus missatges a les xarxes socials per suggerir una seqüència d'activitats turístiques que s'ajustin a diversos contextes i incloguin activitats afins. Per aconseguir-ho, proposem mètodes per identificar els turistes a partir de les seves publicacions a Twitter, identificant les activitats experimentades en aquestes publicacions i perfilant turistes similars en funció dels seus interessos, informació contextual i períodes d'activitat. Aleshores, els perfils d'usuari es combinen amb un algorisme de mineria de regles d'associació per capturar relacions implícites entre els punts d'interès de cada perfil. Finalment, es fa un rànquing de regles i un procés de selecció d'un conjunt d'activitats recomanables. Es va avaluar la precisió de les recomanacions i l'efecte del perfil d'usuari. A més, ordenem el conjunt d'activitats mitjançant un algorisme multi-objectiu per enriquir l'experiència turística. També realitzem una segona fase d'anàlisi dels fluxos turístics a les destinacions que és beneficiós per a les organitzacions de gestió de destinacions, que volen entendre la mobilitat turística. En general, els mètodes i algorismes proposats en aquesta tesi es mostren útils en diversos aspectes dels sistemes de recomanació turística.En los últimos años, los sistemas de recomendación se han vuelto omnipresentes en la web. Muchos servicios web, incluida la transmisión de películas, la búsqueda en la web y el comercio electrónico, utilizan sistemas de recomendación para ayudar a la toma de decisiones. El turismo es una industria altament representada en la web. Hay varios servicios web (e.g. TripAdvisor, Yelp) que se benefician de la inclusión de sistemas recomendadores para ayudar a los turistas a explorar destinos turísticos. Esto ha aumentado la investigación centrada en mejorar los recomendadores turísticos y resolver los principales problemas a los que se enfrentan. Esta tesis propone nuevos algoritmos para sistemas recomendadores turísticos que aprenden las preferencias de los turistas a partir de sus mensajes en redes sociales para sugerir una secuencia de actividades turísticas que se alinean con diversos contextos e incluyen actividades afines. Para lograr esto, proponemos métodos para identificar a los turistas a partir de sus publicaciones en Twitter, identificar las actividades experimentadas en estas publicaciones y perfilar turistas similares en función de sus intereses, contexto información y periodos de actividad. Luego, los perfiles de usuario se combinan con un algoritmo de minería de reglas de asociación para capturar relaciones entre los puntos de interés que aparecen en cada perfil. Finalmente, un proceso de clasificación de reglas y selección de actividades produce un conjunto de actividades recomendables. Se evaluó la precisión de las recomendaciones y el efecto de la elaboración de perfiles de usuario. Ordenamos además el conjunto de actividades utilizando un algoritmo multi-objetivo para enriquecer la experiencia turística. También llevamos a cabo un análisis de los flujos turísticos en los destinos, lo que es beneficioso para las organizaciones de gestión de destinos, que buscan entender la movilidad turística. En general, los métodos y algoritmos propuestos en esta tesis se muestran útiles en varios aspectos de los sistemas de recomendación turística.In recent years, recommender systems have become ubiquitous on the web. Many web services, including movie streaming, web search and e-commerce, use recommender systems to aid human decision-making. Tourism is one industry that is highly represented on the web. There are several web services (e.g. TripAdvisor, Yelp) that benefit from integrating recommender systems to aid tourists in exploring tourism destinations. This has increased research focused on improving tourism recommender systems and solving the main issues they face. This thesis proposes new algorithms for tourism recommender systems that learn tourist preferences from their social media data to suggest a sequence of touristic activities that align with various contexts and include affine activities. To accomplish this, we propose methods for identifying tourists from their frequent Twitter posts, identifying the activities experienced in these posts, and profiling similar tourists based on their interests, contextual information, and activity periods. User profiles are then combined with an association rule mining algorithm for capturing implicit relationships between points of interest apparent in each profile. Finally, a rule ranking and activity selection process produces a set of recommendable activities. The recommendations were evaluated for accuracy and the effect of user profiling. We further order the set of activities using a multi-objective algorithm to enrich the tourist experience. We also carry out a second-stage analysis of tourist flows at destinations which is beneficial to destination management organisations seeking to understand tourist mobility. Overall, the methods and algorithms proposed in this thesis are shown to be useful in various aspects of tourism recommender systems