Socializing the Semantic Gap: A Comparative Survey on Image Tag Assignment, Refinement and Retrieval

Where previous reviews on content-based image retrieval emphasize on what can be seen in an image to bridge the semantic gap, this survey considers what people tag about an image. A comprehensive treatise of three closely linked problems, i.e., image tag assignment, refinement, and tag-based image retrieval is presented. While existing works vary in terms of their targeted tasks and methodology, they rely on the key functionality of tag relevance, i.e. estimating the relevance of a specific tag with respect to the visual content of a given image and its social context. By analyzing what information a specific method exploits to construct its tag relevance function and how such information is exploited, this paper introduces a taxonomy to structure the growing literature, understand the ingredients of the main works, clarify their connections and difference, and recognize their merits and limitations. For a head-to-head comparison between the state-of-the-art, a new experimental protocol is presented, with training sets containing 10k, 100k and 1m images and an evaluation on three test sets, contributed by various research groups. Eleven representative works are implemented and evaluated. Putting all this together, the survey aims to provide an overview of the past and foster progress for the near future.Comment: to appear in ACM Computing Survey

Semantic Interaction in Web-based Retrieval Systems : Adopting Semantic Web Technologies and Social Networking Paradigms for Interacting with Semi-structured Web Data

Existing web retrieval models for exploration and interaction with web data do not take into account semantic information, nor do they allow for new forms of interaction by employing meaningful interaction and navigation metaphors in 2D/3D. This thesis researches means for introducing a semantic dimension into the search and exploration process of web content to enable a significantly positive user experience. Therefore, an inherently dynamic view beyond single concepts and models from semantic information processing, information extraction and human-machine interaction is adopted. Essential tasks for semantic interaction such as semantic annotation, semantic mediation and semantic human-computer interaction were identified and elaborated for two general application scenarios in web retrieval: Web-based Question Answering in a knowledge-based dialogue system and semantic exploration of information spaces in 2D/3D

Enhancing Privacy Management on Social Network Services

Tesis por compendioIn the recent years, social network services, such as Facebook or LinkedIn, have experienced an exponential growth. People enjoy their functionalities, such as sharing photos, finding friends, looking for jobs, and in general, they appreciate the social benefits that social networks provide. However, as using social network has become routine for many people, privacy breaches that may occur in social network services have increased users' concerns. For example, it is easy to find news about people being fired because of something they shared on a social network. To enable people define their privacy settings, service providers employ simple access controls which usually rely exclusively on lists or circles of friends. Although these access controls are easy to configure by average users, research literature points out that they are lacking elements, such as tie strength, that play a key role when users decide what to share and with whom. Additionally, despite the simplicity of current access controls, research on privacy on social media reports that people still struggle to effectively control how their information flows on these services. To provide users with a more robust privacy framework, related literature proposes a new paradigm for access controls based on relationships. In contrast to traditional access controls where permissions are granted based on users and their roles, this paradigm employs social elements such as the relationship between the information owner and potential viewers (e.g., only my siblings can see this photo). Access controls that follow this paradigm provide users with mechanisms for disclosure control that represent more naturally how humans reason about privacy. Furthermore, these access controls can deal with specific issues that social network services present. Specifically, users often share information that concerns many people, especially other members of the social network. In such situations, two or more people can have conflicting privacy preferences; thus, an appropriate sharing policy may not be apparent. These situations are usually identified as multiuser privacy scenarios. Since relationship based access controls are complex for the average social network user, service providers have not adopted them. Therefore, to enable the implementation of such access controls in current social networks, tools and mechanisms that facilitate their use must be provided. To that aim, this thesis makes five contributions: (1) a review of related research on privacy management on social networks that identifies pressing challenges in the field, (2) BFF, a tool for eliciting automatically tie strength and user communities, (3) a new access control that employs communities, individual identifiers, tie strength, and content tags, (4) a novel model for representing and reasoning about multiuser privacy scenarios, employing three types of features: contextual factors, user preferences, and user arguments; and, (5) Muppet, a tool that recommends sharing policies in multiuser privacy scenarios.En los últimos años, los servicios de redes sociales, como Facebook o LinkedIn, han experimentado un crecimiento exponencial. Los usuarios valoran positivamente sus muchas funcionalidades tales como compartir fotos, o búsqueda de amigos y trabajo. En general, los usuarios aprecian los beneficios que las redes sociales les aportan. Sin embargo, mientras el uso de redes sociales se ha convertido en rutina para mucha gente, brechas de privacidad que pueden ocurrir en redes sociales han aumentado los recelos de los usuarios. Por ejemplo, es sencillo encontrar en las noticias casos sobre personas que han perdido su empleo debido a algo que compartieron en una red social. Para facilitar la definición de los ajustes de privacidad, los proveedores de servicios emplean controles de acceso sencillos que normalmente se basan, de forma exclusiva, en listas o círculos de amigos. Aunque estos controles de acceso son fáciles de configurar por un usuario medio, investigaciones recientes indican que éstos carecen de elementos tales como la intensidad de los vínculos personales, que juegan un papel clave en cómo los usuarios deciden qué compartir y con quién. Además, a pesar de la simplicidad de los controles de acceso, investigaciones sobre privacidad en redes sociales señalan que los usuarios han de esforzarse para controlar de forma efectiva como su información fluye en estos servicios. Para ofrecer a los usuarios un marco de privacidad más robusto, trabajos recientes proponen un nuevo paradigma para controles de acceso basado en relaciones. A diferencia de los controles de acceso tradicionales donde los permisos se otorgan en base a usuarios y sus roles, este paradigma emplea elementos sociales como la relación entre el propietario de la información y su audiencia potencial (por ejemplo, sólo mis hermanos pueden ver la foto). Los controles de acceso que siguen este paradigma ofrecen a los usuarios mecanismos para el control de la privacidad que representan de una forma más natural como los humanos razonan sobre cuestiones de privacidad. Además, estos controles de acceso pueden lidiar con problemáticas específicas que presentan las redes sociales. Específicamente, los usuarios comparten de forma habitual información que atañe a muchas personas, especialmente a otros miembros de la red social. En tales situaciones, dos o más personas pueden tener preferencias de privacidad que entran en conflicto. Cuando esto ocurre, no hay una configuración correcta de privacidad que sea evidente. Estas situaciones son normalmente identificadas como escenarios de privacidad multiusuario. Dado que los controles de acceso basados en relaciones son complejos para el usuario promedio de redes sociales, los proveedores de servicios no los han adoptado. Por lo tanto, para permitir la implementación de tales controles de acceso en redes sociales actuales, es necesario que se ofrezcan herramientas y mecanismos que faciliten su uso. En este sentido, esta tesis presenta cinco contribuciones: (1) una revisión del estado del arte en manejo de privacidad en redes sociales que permite identificar los retos más importantes en el campo, (2) BFF, una herramienta para obtener automáticamente la intensidad de los vínculos personales y las comunidades de usuarios, (3) un nuevo control de acceso que emplea comunidades, identificadores individuales, la intensidad de los vínculos personales, y etiquetas de contenido, (4) un modelo novedoso para representar y razonar sobre escenarios de privacidad multiusario que emplea tres tipos de características: factores contextuales, preferencias de usuario, y argumentos de usuario; y, (5) Muppet, una herramienta que recomienda configuraciones de privacidad en escenarios de privacidad multiusuario.En els darrers anys, els servicis de xarxes socials, com Facebook o LinkedIn, han experimentat un creixement exponencial. Els usuaris valoren positivament les seues variades funcionalitats com la compartició de fotos o la cerca d'amics i treball. En general, els usuaris aprecien els beneficis que les xarxes socials els aporten. No obstant això, mentre l'ús de les xarxes socials s'ha convertit en rutina per a molta gent, bretxes de privacitat que poden ocórrer en xarxes socials han augmentat els recels dels usuaris. Per exemple, és senzill trobar notícies sobre persones que han perdut el seu treball per alguna cosa que compartiren a una xarxa social. Per facilitar la definició dels ajustos de privacitat, els proveïdors de servicis empren controls d'accés senzills que normalment es basen, de forma exclusiva, en llistes o cercles d'amics. Encara que aquests controls d'accés són fàcils d'emprar per a un usuari mitjà, investigacions recents indiquen que aquests manquen elements com la força dels vincles personals, que juguen un paper clau en com els usuaris decideixen què compartir i amb qui. A més a més, malgrat la simplicitat dels controls d'accés, investigacions sobre privacitat en xarxes socials revelen que els usuaris han d'esforçar-se per a controlar de forma efectiva com fluix la seua informació en aquests servicis. Per a oferir als usuaris un marc de privacitat més robust, treballs recents proposen un nou paradigma per a controls d'accés basat en relacions. A diferència dels controls d'accés tradicionals on els permisos s'atorguen segons usuaris i els seus rols, aquest paradigma empra elements socials com la relació entre el propietari de la informació i la seua audiència potencial (per exemple, sols els meus germans poden veure aquesta foto). Els controls d'accés que segueixen aquest paradigma ofereixen als usuaris mecanismes per al control de la privacitat que representen d'una forma més natural com els humans raonen sobre la privacitat. A més a més, aquests controls d'accés poden resoldre problemàtiques específiques que presenten les xarxes socials. Específicament, els usuaris comparteixen de forma habitual informació que concerneix moltes persones, especialment a altres membres de la xarxa social. En aquestes situacions, dues o més persones poden tindre preferències de privacitat que entren en conflicte. Quan açò ocorre, no hi ha una configuració de privacitat correcta que siga evident. Aquestes situacions són normalment identificades com escenaris de privacitat multiusari. Donat que els controls d'accés basats en relacions són complexos per a l'usuari mitjà de xarxes socials, els proveïdors de servicis no els han adoptat. Per tant, per a permetre la implementació d'aquests controls d'accés en xarxes socials actuals, és necessari oferir ferramentes i mecanismes que faciliten el seu ús. En aquest sentit, aquesta tesi presenta cinc contribucions: (1) una revisió de l'estat de l'art en maneig de privacitat en xarxes socials que permet identificar els reptes més importants en el camp, (2) BFF, una ferramenta per a obtenir automàticament la força dels vincles personals i les comunitats d'usuaris, (3) un nou control d'accés que empra comunitats, identificadors individuals, força dels vincles personals, i etiquetes de contingut, (4) un model nou per a representar i raonar sobre escenaris de privacitat multiusari que empra tres tipus de característiques: factors contextuals, preferències d'usuari, i arguments d'usuaris; i, (5) Muppet, una ferramenta que recomana configuracions de privacitat en escenaris de privacitat multiusuari.López Fogués, R. (2017). Enhancing Privacy Management on Social Network Services [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/85978TESISCompendi

Image Understanding by Socializing the Semantic Gap

Several technological developments like the Internet, mobile devices and Social Networks have spurred the sharing of images in unprecedented volumes, making tagging and commenting a common habit. Despite the recent progress in image analysis, the problem of Semantic Gap still hinders machines in fully understand the rich semantic of a shared photo. In this book, we tackle this problem by exploiting social network contributions. A comprehensive treatise of three linked problems on image annotation is presented, with a novel experimental protocol used to test eleven state-of-the-art methods. Three novel approaches to annotate, under stand the sentiment and predict the popularity of an image are presented. We conclude with the many challenges and opportunities ahead for the multimedia community

Semantically-enhanced image tagging system

In multimedia databases, data are images, audio, video, texts, etc. Research interests in these types of databases have increased in the last decade or so, especially with the advent of the Internet and Semantic Web. Fundamental research issues vary from unified data modelling, retrieval of data items and dynamic nature of updates. The thesis builds on findings in Semantic Web and retrieval techniques and explores novel tagging methods for identifying data items. Tagging systems have become popular which enable the users to add tags to Internet resources such as images, video and audio to make them more manageable. Collaborative tagging is concerned with the relationship between people and resources. Most of these resources have metadata in machine processable format and enable users to use free- text keywords (so-called tags) as search techniques. This research references some tagging systems, e.g. Flicker, delicious and myweb2.0. The limitation with such techniques includes polysemy (one word and different meaning), synonymy (different words and one meaning), different lexical forms (singular, plural, and conjugated words) and misspelling errors or alternate spellings. The work presented in this thesis introduces semantic characterization of web resources that describes the structure and organization of tagging, aiming to extend the existing Multimedia Query using similarity measures to cater for collaborative tagging. In addition, we discuss the semantic difficulties of tagging systems, suggesting improvements in their accuracies. The scope of our work is classified as follows: (i) Increase the accuracy and confidence of multimedia tagging systems. (ii) Increase the similarity measures of images by integrating varieties of measures. To address the first shortcoming, we use the WordNet based on a tagging system for social sharing and retrieval of images as a semantic lingual ontology resource. For the second shortcoming we use the similarity measures in different ways to recognise the multimedia tagging system. Fundamental to our work is the novel information model that we have constructed for our computation. This is based on the fact that an image is a rich object that can be characterised and formulated in n-dimensions, each dimension contains valuable information that will help in increasing the accuracy of the search. For example an image of a tree in a forest contains more information than an image of the same tree but in a different environment. In this thesis we characterise a data item (an image) by a primary description, followed by n-secondary descriptions. As n increases, the accuracy of the search improves. We give various techniques to analyse data and its associated query. To increase the accuracy of the tagging system we have performed different experiments on many images using similarity measures and various techniques from VoI (Value of Information). The findings have shown the linkage/integration between similarity measures and that VoI improves searches and helps/guides a tagger in choosing the most adequate of tags

The role of context in image annotation and recommendation

With the rise of smart phones, lifelogging devices (e.g. Google Glass) and popularity of image sharing websites (e.g. Flickr), users are capturing and sharing every aspect of their life online producing a wealth of visual content. Of these uploaded images, the majority are poorly annotated or exist in complete semantic isolation making the process of building retrieval systems difficult as one must firstly understand the meaning of an image in order to retrieve it. To alleviate this problem, many image sharing websites offer manual annotation tools which allow the user to “tag” their photos, however, these techniques are laborious and as a result have been poorly adopted; Sigurbjörnsson and van Zwol (2008) showed that 64% of images uploaded to Flickr are annotated with < 4 tags. Due to this, an entire body of research has focused on the automatic annotation of images (Hanbury, 2008; Smeulders et al., 2000; Zhang et al., 2012a) where one attempts to bridge the semantic gap between an image’s appearance and meaning e.g. the objects present. Despite two decades of research the semantic gap still largely exists and as a result automatic annotation models often offer unsatisfactory performance for industrial implementation. Further, these techniques can only annotate what they see, thus ignoring the “bigger picture” surrounding an image (e.g. its location, the event, the people present etc). Much work has therefore focused on building photo tag recommendation (PTR) methods which aid the user in the annotation process by suggesting tags related to those already present. These works have mainly focused on computing relationships between tags based on historical images e.g. that NY and timessquare co-exist in many images and are therefore highly correlated. However, tags are inherently noisy, sparse and ill-defined often resulting in poor PTR accuracy e.g. does NY refer to New York or New Year? This thesis proposes the exploitation of an image’s context which, unlike textual evidences, is always present, in order to alleviate this ambiguity in the tag recommendation process. Specifically we exploit the “what, who, where, when and how” of the image capture process in order to complement textual evidences in various photo tag recommendation and retrieval scenarios. In part II, we combine text, content-based (e.g. # of faces present) and contextual (e.g. day-of-the-week taken) signals for tag recommendation purposes, achieving up to a 75% improvement to precision@5 in comparison to a text-only TF-IDF baseline. We then consider external knowledge sources (i.e. Wikipedia & Twitter) as an alternative to (slower moving) Flickr in order to build recommendation models on, showing that similar accuracy could be achieved on these faster moving, yet entirely textual, datasets. In part II, we also highlight the merits of diversifying tag recommendation lists before discussing at length various problems with existing automatic image annotation and photo tag recommendation evaluation collections. In part III, we propose three new image retrieval scenarios, namely “visual event summarisation”, “image popularity prediction” and “lifelog summarisation”. In the first scenario, we attempt to produce a rank of relevant and diverse images for various news events by (i) removing irrelevant images such memes and visual duplicates (ii) before semantically clustering images based on the tweets in which they were originally posted. Using this approach, we were able to achieve over 50% precision for images in the top 5 ranks. In the second retrieval scenario, we show that by combining contextual and content-based features from images, we are able to predict if it will become “popular” (or not) with 74% accuracy, using an SVM classifier. Finally, in chapter 9 we employ blur detection and perceptual-hash clustering in order to remove noisy images from lifelogs, before combining visual and geo-temporal signals in order to capture a user’s “key moments” within their day. We believe that the results of this thesis show an important step towards building effective image retrieval models when there lacks sufficient textual content (i.e. a cold start)