Mixing Modalities of 3D Sketching and Speech for Interactive Model Retrieval in Virtual Reality

Sketch and speech are intuitive interaction methods that convey complementary information and have been independently used for 3D model retrieval in virtual environments. While sketch has been shown to be an effective retrieval method, not all collections are easily navigable using this modality alone. We design a new challenging database for sketch comprised of 3D chairs where each of the components (arms, legs, seat, back) are independently colored. To overcome this, we implement a multimodal interface for querying 3D model databases within a virtual environment. We base the sketch on the state-of-the-art for 3D Sketch Retrieval, and use a Wizard-of-Oz style experiment to process the voice input. In this way, we avoid the complexities of natural language processing which frequently requires fine-tuning to be robust. We conduct two user studies and show that hybrid search strategies emerge from the combination of interactions, fostering the advantages provided by both modalities

DeepSketch2Face: A Deep Learning Based Sketching System for 3D Face and Caricature Modeling

Face modeling has been paid much attention in the field of visual computing. There exist many scenarios, including cartoon characters, avatars for social media, 3D face caricatures as well as face-related art and design, where low-cost interactive face modeling is a popular approach especially among amateur users. In this paper, we propose a deep learning based sketching system for 3D face and caricature modeling. This system has a labor-efficient sketching interface, that allows the user to draw freehand imprecise yet expressive 2D lines representing the contours of facial features. A novel CNN based deep regression network is designed for inferring 3D face models from 2D sketches. Our network fuses both CNN and shape based features of the input sketch, and has two independent branches of fully connected layers generating independent subsets of coefficients for a bilinear face representation. Our system also supports gesture based interactions for users to further manipulate initial face models. Both user studies and numerical results indicate that our sketching system can help users create face models quickly and effectively. A significantly expanded face database with diverse identities, expressions and levels of exaggeration is constructed to promote further research and evaluation of face modeling techniques.Comment: 12 pages, 16 figures, to appear in SIGGRAPH 201

Sketch-based Queries in Mobile GIS-Environments

Recent achievements in the field of mobile computing and wireless communication promise data retrieval anywhere and anytime. This development provided the basis to expand GIs technology to handheld devices, such as personal digital assistants (PDAs). Although traditional GIs technology is well suited for desktop workstations, it needs to be adapted in order to satisfy the requirements of users using handheld computing devices. This adaptation is necessary because the usability of traditional GISs depends on characteristics of desktop computers, such as their relatively large user interfaces (e.g., displays, keyboards, pointing devices), considerable computing resources (i.e., CPU, memory, storage, operating systems), and high bandwidth network connectivity. Small devices possess few of these characteristics, hence, requiring new and efficient methods for interaction with spatial databases. We propose a concept that supports sketch-based querying in mobile GIs environments. This concept combines newest techniques for spatial querying and mobile technologies. Such a combination is beneficial for users because it allows them to formulate queries by drawing the desired configuration with a pen on the touch-sensitive PDA screen, and consequently avoids typing complex statements in some SQL-like query language. Client-server architectures in mobile environments are characterized by low and fluctuating bandwidth, and by frequent disconnections. We discuss client-server strategies in mobile environments, suggest an adaptive client-server architecture for geomobile querying, and analyze the performance. It is shown that adaptation to the mobile environment is necessary in order to ensure efficiency of geo-mobile queries

a.SCatch: semantic structure for architectural floor plan retrieval

Architects’ daily routine involves working with drawings. They use either a pen or a computer to sketch out their ideas or to do a drawing to scale. We therefore propose the use of a sketch-based approach when using the floor plan repository for queries. This enables the user of the system to sketch a schematic abstraction of a floor plan and search for floor plans that are structurally similar. We also propose the use of a visual query language, and a semantic structure as put forward by Langenhan. An algorithm extracts the semantic structure sketched by the architect on DFKI’s Touch& Write table and compares the structure of the sketch with that of those from the floor plan repository. The a.SCatch system enables the user to access knowledge from past projects easily. Based on CBR strategies and shape detection technologies, a sketch-based retrieval gives access to a semantic floor plan repository. Furthermore, details of a prototypical application which allows semantic structure to be extracted from image data and put into the repository semi-automatically are provided