Perceptually Guided Photo Retargeting

We propose perceptually guided photo retargeting, which shrinks a photo by simulating a human's process of sequentially perceiving visually/semantically important regions in a photo. In particular, we first project the local features (graphlets in this paper) onto a semantic space, wherein visual cues such as global spatial layout and rough geometric context are exploited. Thereafter, a sparsity-constrained learning algorithm is derived to select semantically representative graphlets of a photo, and the selecting process can be interpreted by a path which simulates how a human actively perceives semantics in a photo. Furthermore, we learn the prior distribution of such active graphlet paths (AGPs) from training photos that are marked as esthetically pleasing by multiple users. The learned priors enforce the corresponding AGP of a retargeted photo to be maximally similar to those from the training photos. On top of the retargeting model, we further design an online learning scheme to incrementally update the model with new photos that are esthetically pleasing. The online update module makes the algorithm less dependent on the number and contents of the initial training data. Experimental results show that: 1) the proposed AGP is over 90% consistent with human gaze shifting path, as verified by the eye-tracking data, and 2) the retargeting algorithm outperforms its competitors significantly, as AGP is more indicative of photo esthetics than conventional saliency maps

Computational intelligence approaches to robotics, automation, and control [Volume guest editors]

No abstract available

Visual saliency computation for image analysis

Visual saliency computation is about detecting and understanding salient regions and elements in a visual scene. Algorithms for visual saliency computation can give clues to where people will look in images, what objects are visually prominent in a scene, etc. Such algorithms could be useful in a wide range of applications in computer vision and graphics. In this thesis, we study the following visual saliency computation problems. 1) Eye Fixation Prediction. Eye fixation prediction aims to predict where people look in a visual scene. For this problem, we propose a Boolean Map Saliency (BMS) model which leverages the global surroundedness cue using a Boolean map representation. We draw a theoretic connection between BMS and the Minimum Barrier Distance (MBD) transform to provide insight into our algorithm. Experiment results show that BMS compares favorably with state-of-the-art methods on seven benchmark datasets. 2) Salient Region Detection. Salient region detection entails computing a saliency map that highlights the regions of dominant objects in a scene. We propose a salient region detection method based on the Minimum Barrier Distance (MBD) transform. We present a fast approximate MBD transform algorithm with an error bound analysis. Powered by this fast MBD transform algorithm, our method can run at about 80 FPS and achieve state-of-the-art performance on four benchmark datasets. 3) Salient Object Detection. Salient object detection targets at localizing each salient object instance in an image. We propose a method using a Convolutional Neural Network (CNN) model for proposal generation and a novel subset optimization formulation for bounding box filtering. In experiments, our subset optimization formulation consistently outperforms heuristic bounding box filtering baselines, such as Non-maximum Suppression, and our method substantially outperforms previous methods on three challenging datasets. 4) Salient Object Subitizing. We propose a new visual saliency computation task, called Salient Object Subitizing, which is to predict the existence and the number of salient objects in an image using holistic cues. To this end, we present an image dataset of about 14K everyday images which are annotated using an online crowdsourcing marketplace. We show that an end-to-end trained CNN subitizing model can achieve promising performance without requiring any localization process. A method is proposed to further improve the training of the CNN subitizing model by leveraging synthetic images. 5) Top-down Saliency Detection. Unlike the aforementioned tasks, top-down saliency detection entails generating task-specific saliency maps. We propose a weakly supervised top-down saliency detection approach by modeling the top-down attention of a CNN image classifier. We propose Excitation Backprop and the concept of contrastive attention to generate highly discriminative top-down saliency maps. Our top-down saliency detection method achieves superior performance in weakly supervised localization tasks on challenging datasets. The usefulness of our method is further validated in the text-to-region association task, where our method provides state-of-the-art performance using only weakly labeled web images for training

Iconic architecture through the lens of Instagram: the case studies of the Guggenheim Museum, Bilbao and the Dongdaemun Design Plaza, Seoul

Architecture has played an enormous role in the branding of cities, initially through cultural institutions such as museums, which have become the preferred platform for the expression of iconic architecture to boost the image of a city’s modernity and economic prosperity, and to express its civic pride. In recent years the seemingly endless potential of social media has allowed the consumption of architecture to surpass the boundaries of space and time. The instant image sharing and dissemination of Instagrammably photogenic iconic architecture has made the notion of ‘iconicity’ more questionable than it might have been before the social media era. This research aims to explore the manner in which contemporary iconic architecture is represented in social media, with a specific focus on the manner in which such architectural imagery moulds ‘iconicity’ in architecture; in doing so, it investigates the ways in which city image is incorporated into the social imagery of architecture. Using the two case studies of Frank Ghery’s Guggenheim Museum in Bilbao and Zaha Hadid’s Dongdaemun Design Plaza and Park in Seoul, the thesis scrutinises user-generated photographic images and accompanying textual descriptions, which were downloaded from Instagram. The empirical work involves a two-part multi-method approach combining visual content analysis and discourse analysis, using an adaptation of Panofsky’s Iconology, which was borrowed from art history. A general picture of the representational practices of Instagram images was gained through content analysis; this is followed by qualitative readings of individual images using Panofsky’s iconographic-iconological method. The results demonstrate that there are key elements that convey architectural iconicity in Instagram images. These include: (a) the heightened aesthetics of image-taking through the maximisation of aesthetic value in the portrayal of a building; (b) verbal texts alongside an image, which deliver information on the building; and (c) geographic associations through geo-tagging and hashtagging, and textual components, such as a caption and comments. The findings further indicate that, given that a majority of images are depicted in relation to architectural context, this context, in other words, the place in which a building is situated, is essential for the reception and perception of iconicity in the building. The present study is cross-disciplinary in nature, which serves as an important contribution to academic research into place branding by bringing together architecture, city branding, and social media. This is the first time that the Panofsky model of iconology has been applied to the field of place branding

VISUAL SALIENCY ANALYSIS AND APPLICATIONS

Ph.DDOCTOR OF PHILOSOPH

Between Texts and Cities

TEXTS+CITIES explores the relation between texts and urban spaces in contemporary culture and society. Cities have often been compared to palimpsests, their streets, buildings, and subways pleated, crumpled, written and rewritten over and over again: as material texts, poïesis. What is at stake in this conflation of city and text? How do urban spaces relate to artistic, political, or economic texts and ideologies? What transformations occur between the designing of urban spaces, and the building and eventual inhabiting of those spaces? TEXTS+CITIES aims to bring together scholars and practitioners within an interdisciplinary range of social sciences, humanities, art, design, and media to reflect on ways of producing, reproducing, and experiencing the urban.Peer reviewe

Computational intelligence approaches to robotics, automation, and control [Volume guest editors]

No abstract available