Text-Guided Scene Sketch-to-Photo Synthesis

We propose a method for scene-level sketch-to-photo synthesis with text guidance. Although object-level sketch-to-photo synthesis has been widely studied, whole-scene synthesis is still challenging without reference photos that adequately reflect the target style. To this end, we leverage knowledge from recent large-scale pre-trained generative models, resulting in text-guided sketch-to-photo synthesis without the need for reference images. To train our model, we use self-supervised learning from a set of photographs. Specifically, we use a pre-trained edge detector that maps both color and sketch images into a standardized edge domain, which reduces the gap between photograph-based edge images (during training) and hand-drawn sketch images (during inference). We implement our method by fine-tuning a latent diffusion model (i.e., Stable Diffusion) with sketch and text conditions. Experiments show that the proposed method translates original sketch images that are not extracted from color images into photos with compelling visual quality

Validation of an Aesthetic Assessment System for Commercial Tasks

[Abstract] Automatic prediction of the aesthetic value of images has received increasing attention in recent years. This is due, on the one hand, to the potential impact that predicting the aesthetic value has on practical applications. Even so, it remains a difficult task given the subjectivity and complexity of the problem. An image aesthetics assessment system was developed in recent years by our research group. In this work, its potential to be applied in commercial tasks is tested. With this objective, a set of three portals and three real estate agencies in Spain were taken as case studies. Images of their websites were taken to build the experimental dataset and a validation method was developed to test their original order with another proposed one according to their aesthetic value. So, in this new order, the images that have the high aesthetic score by the AI system will occupy the first positions of the portal. Relevant results were obtained, with an average increase of 52.54% in the number of clicks on the ads, in the experiment with Real Estate portals. A statistical analysis prove that there is a significant difference in the number of clicks after selecting the images with the AI system.This work is supported by the General Directorate of Culture, Education and University Management of Xunta de Galicia (Ref. ED431D 201716), Competitive Reference Groups (Ref. ED431C 201849) and Ministry of Science and Innovation project Society challenges (Ref. PID2020-118362RB-I00). We also wish to acknowledge the support received from the Centro de Investigación de Galicia “CITIC”, funded by Xunta de Galicia and the European Union (European Regional Development Fund- Galicia 2014-2020 Program), by grant ED431G 2019/01Xunta de Galicia; ED431D 201716Xunta de Galicia; ED431C 201849Xunta de Galicia; ED431G 2019/0

Puffy: A Step-by-step Guide to Craft Bio-inspired Artifacts with Interactive Materiality

A rising number of HCI scholars have begun to use materiality as a starting point for exploring the design's potential and restrictions. Despite the theoretical flourishing, the practical design process and instruction for beginner practitioners are still in scarcity. We leveraged the pictorial format to illustrate our crafting process of Puffy, a bio-inspired artifact that features a cilia-mimetic surface expressing anthropomorphic qualities through shape changes. Our approach consists of three key activities (i.e., analysis, synthesis, and detailing) interlaced recursively throughout the journey. Using this approach, we analyzed different input sources, synthesized peers' critiques and self-reflection, and detailed the designed experience with iterative prototypes. Building on a reflective analysis of our approach, we concluded with a set of practical implications and design recommendations to inform other practitioners to initiate their investigations in interactive materiality.Comment: 17th International Conference On Tangible Embedded And Embodied Interactio

Enhancing travel experience with the combination of information visualization, situation awareness, and distributed cognition

With the new forms of travel introduced by new technologies of transportation and communication, a satisfied travel experience could be affected by various factors before and during a trip. Especially for road trips, traveling by car provides freedom on time control while leading to more possibilities of rescheduling initial plans made under time constraints. When overwhelmed with the need for changed travel context to avoid unexpected events that will require a serious change of initial plans, travelers need to find and access helpful contextual information quickly. This is a context-related decision making process that requires amplifying human situation awareness and supporting distributed cognition, since travel information offers multiple choices. To solve this problem, I applied information visualization as the main design solution. When comparing it with a traditional representation of lists, information visualization displays the advantages of visual representation of abstract data to clarify and depict the information and amplify cognition while improving travel experience intuitively in the domain of user experience design. Therefore in this thesis I will address the approach of implementing recontextualized situation awareness, distributed cognition, and information visualization in a travel-aid system. By using both theoretical and practical design perspectives, I will discuss how to enhance travel experience with represented contextual information that users desire or expect before and during a road trip. I will also explore the new values of this design with strategic business support. Additionally, after conducting research and analysis on existing interaction design parts, I selected a smartphone app to serve as a proper platform with connected multifunctions. Briefly, I begin the thesis with a review of previous theories and aspects of travel planning, information visualization as it relates to travel, situation awareness, and distributed cognition in the design context and related smartphone apps. Then I discuss the process of identifying the specific issues to be solved or improved with a preliminary research of empirical study, followed by an interview, online survey, insights synthesis, and business model design. After a visual-system design was developed, heuristic evaluation was employed to assess the outcome. Lastly, a new round of refined design results is introduced based on outcomes of the evaluation

SimpSON: Simplifying Photo Cleanup with Single-Click Distracting Object Segmentation Network

In photo editing, it is common practice to remove visual distractions to improve the overall image quality and highlight the primary subject. However, manually selecting and removing these small and dense distracting regions can be a laborious and time-consuming task. In this paper, we propose an interactive distractor selection method that is optimized to achieve the task with just a single click. Our method surpasses the precision and recall achieved by the traditional method of running panoptic segmentation and then selecting the segments containing the clicks. We also showcase how a transformer-based module can be used to identify more distracting regions similar to the user's click position. Our experiments demonstrate that the model can effectively and accurately segment unknown distracting objects interactively and in groups. By significantly simplifying the photo cleaning and retouching process, our proposed model provides inspiration for exploring rare object segmentation and group selection with a single click.Comment: CVPR 2023. Project link: https://simpson-cvpr23.github.i

Progressive Joint Low-light Enhancement and Noise Removal for Raw Images

Low-light imaging on mobile devices is typically challenging due to insufficient incident light coming through the relatively small aperture, resulting in a low signal-to-noise ratio. Most of the previous works on low-light image processing focus either only on a single task such as illumination adjustment, color enhancement, or noise removal; or on a joint illumination adjustment and denoising task that heavily relies on short-long exposure image pairs collected from specific camera models, and thus these approaches are less practical and generalizable in real-world settings where camera-specific joint enhancement and restoration is required. To tackle this problem, in this paper, we propose a low-light image processing framework that performs joint illumination adjustment, color enhancement, and denoising. Considering the difficulty in model-specific data collection and the ultra-high definition of the captured images, we design two branches: a coefficient estimation branch as well as a joint enhancement and denoising branch. The coefficient estimation branch works in a low-resolution space and predicts the coefficients for enhancement via bilateral learning, whereas the joint enhancement and denoising branch works in a full-resolution space and progressively performs joint enhancement and denoising. In contrast to existing methods, our framework does not need to recollect massive data when being adapted to another camera model, which significantly reduces the efforts required to fine-tune our approach for practical usage. Through extensive experiments, we demonstrate its great potential in real-world low-light imaging applications when compared with current state-of-the-art methods