DeepSketch2Face: A Deep Learning Based Sketching System for 3D Face and Caricature Modeling

Face modeling has been paid much attention in the field of visual computing. There exist many scenarios, including cartoon characters, avatars for social media, 3D face caricatures as well as face-related art and design, where low-cost interactive face modeling is a popular approach especially among amateur users. In this paper, we propose a deep learning based sketching system for 3D face and caricature modeling. This system has a labor-efficient sketching interface, that allows the user to draw freehand imprecise yet expressive 2D lines representing the contours of facial features. A novel CNN based deep regression network is designed for inferring 3D face models from 2D sketches. Our network fuses both CNN and shape based features of the input sketch, and has two independent branches of fully connected layers generating independent subsets of coefficients for a bilinear face representation. Our system also supports gesture based interactions for users to further manipulate initial face models. Both user studies and numerical results indicate that our sketching system can help users create face models quickly and effectively. A significantly expanded face database with diverse identities, expressions and levels of exaggeration is constructed to promote further research and evaluation of face modeling techniques.Comment: 12 pages, 16 figures, to appear in SIGGRAPH 201

Virtual 3D Map for Library User. (vSearch)

This report describes the architecture and preliminary design of a prototype system to enhance services/tasks of the current online digital library system through the integration of virtual 3d map. The aim for a virtual map is to further assist the end user on location a book's physical location in the library. Customarily, an online catalogue search for books/journals/magazine would provide the user with details of the books/journals/magazines including information on books' locations; none have shown the user how to get to the exact location visually. This capability only can be done by integrating a virtual 3D map into a simulation of current online system. The main purpose of adding the virtual element is to provide the user with the interactive virtual environment that can help user navigate through the library environment to find desired location. Instead of knowing where the material location is, user also will get to know how to get 'there' and which path can aid the user to get 'there '. An initial survey has been done on library users' reveals that locating the library material takes up much of their time. A virtual 3d map was proposed to be incorporated into the system to reduce the amount of time to acquire the library material. Modeling the virtual 3D map into current online system will make for an extremely useful resource. User will be able to access the 3D map as a Flash application via the online system. This will allow users to easily locate specific material within the actual library, to determine the best way to get to a location and possibly to get suggestions on what route to take. The major challenge of the design is the integration of a simulated database of the current system to the virtual 3D map applications. Other enhancements done in current library includes a more user friendly interface. Keywords : vSearch, virtual navigation, navigation aid

Mixing Modalities of 3D Sketching and Speech for Interactive Model Retrieval in Virtual Reality

Sketch and speech are intuitive interaction methods that convey complementary information and have been independently used for 3D model retrieval in virtual environments. While sketch has been shown to be an effective retrieval method, not all collections are easily navigable using this modality alone. We design a new challenging database for sketch comprised of 3D chairs where each of the components (arms, legs, seat, back) are independently colored. To overcome this, we implement a multimodal interface for querying 3D model databases within a virtual environment. We base the sketch on the state-of-the-art for 3D Sketch Retrieval, and use a Wizard-of-Oz style experiment to process the voice input. In this way, we avoid the complexities of natural language processing which frequently requires fine-tuning to be robust. We conduct two user studies and show that hybrid search strategies emerge from the combination of interactions, fostering the advantages provided by both modalities

DualMotion: Global-to-Local Casual Motion Design for Character Animations

Animating 3D characters using motion capture data requires basic expertise and manual labor. To support the creativity of animation design and make it easier for common users, we present a sketch-based interface DualMotion, with rough sketches as input for designing daily-life animations of characters, such as walking and jumping.Our approach enables to combine global motions of lower limbs and the local motion of the upper limbs in a database by utilizing a two-stage design strategy. Users are allowed to design a motion by starting with drawing a rough trajectory of a body/lower limb movement in the global design stage. The upper limb motions are then designed by drawing several more relative motion trajectories in the local design stage. We conduct a user study and verify the effectiveness and convenience of the proposed system in creative activities.Comment: 10 pages, 10 figures, under submission, video is here https://youtu.be/-tk8q8LSiL

SuperSweet—a resource on natural and artificial sweetening agents

A vast number of sweet tasting molecules are known, encompassing small compounds, carbohydrates, d-amino acids and large proteins. Carbohydrates play a particularly big role in human diet. The replacement of sugars in food with artificial sweeteners is common and is a general approach to prevent cavities, obesity and associated diseases such as diabetes and hyperlipidemia. Knowledge about the molecular basis of taste may reveal new strategies to overcome diet-induced diseases. In this context, the design of safe, low-calorie sweeteners is particularly important. Here, we provide a comprehensive collection of carbohydrates, artificial sweeteners and other sweet tasting agents like proteins and peptides. Additionally, structural information and properties such as number of calories, therapeutic annotations and a sweetness-index are stored in SuperSweet. Currently, the database consists of more than 8000 sweet molecules. Moreover, the database provides a modeled 3D structure of the sweet taste receptor and binding poses of the small sweet molecules. These binding poses provide hints for the design of new sweeteners. A user-friendly graphical interface allows similarity searching, visualization of docked sweeteners into the receptor etc. A sweetener classification tree and browsing features allow quick requests to be made to the database. The database is freely available at: http://bioinformatics.charite.de/sweet/

XUIMS: the X-Window User Interface Management System at CERN

The CERN X-Window User Interface Management System (XUIMS) is a modular and highly configurable software development environment allowing the interactive design, prototyping, and production of OSF/Motif Human Computer Interfaces (HCI). Fully compliant with the X11R5 and OSF/Motif industry standards, XUIMS covers complex software areas like the development of schematics, the visualization and on-line interactions with 2D and 3D scientific data, the display of relational database data, and the direct access to CERN SPS and LEP accelerators equipment. The guarantee of consistency across the applications and the encapsulation of complex functionality in re-usable and user-friendly components has also been implemented through the development of home made graphical objects (widgets) and templates. The XUIMS environment is built with commercial software products integrated in the CERN SPS and LEP controls infrastructure with a very limited home-made effort. Productivity and quality have been improved through less coding and better HCI prototyping

Application of Machine Learning within Visual Content Production

We are living in an era where digital content is being produced at a dazzling pace. The heterogeneity of contents and contexts is so varied that a numerous amount of applications have been created to respond to people and market demands. The visual content production pipeline is the generalisation of the process that allows a content editor to create and evaluate their product, such as a video, an image, a 3D model, etc. Such data is then displayed on one or more devices such as TVs, PC monitors, virtual reality head-mounted displays, tablets, mobiles, or even smartwatches. Content creation can be simple as clicking a button to film a video and then share it into a social network, or complex as managing a dense user interface full of parameters by using keyboard and mouse to generate a realistic 3D model for a VR game. In this second example, such sophistication results in a steep learning curve for beginner-level users. In contrast, expert users regularly need to refine their skills via expensive lessons, time-consuming tutorials, or experience. Thus, user interaction plays an essential role in the diffusion of content creation software, primarily when it is targeted to untrained people. In particular, with the fast spread of virtual reality devices into the consumer market, new opportunities for designing reliable and intuitive interfaces have been created. Such new interactions need to take a step beyond the point and click interaction typical of the 2D desktop environment. The interactions need to be smart, intuitive and reliable, to interpret 3D gestures and therefore, more accurate algorithms are needed to recognise patterns. In recent years, machine learning and in particular deep learning have achieved outstanding results in many branches of computer science, such as computer graphics and human-computer interface, outperforming algorithms that were considered state of the art, however, there are only fleeting efforts to translate this into virtual reality. In this thesis, we seek to apply and take advantage of deep learning models to two different content production pipeline areas embracing the following subjects of interest: advanced methods for user interaction and visual quality assessment. First, we focus on 3D sketching to retrieve models from an extensive database of complex geometries and textures, while the user is immersed in a virtual environment. We explore both 2D and 3D strokes as tools for model retrieval in VR. Therefore, we implement a novel system for improving accuracy in searching for a 3D model. We contribute an efficient method to describe models through 3D sketch via an iterative descriptor generation, focusing both on accuracy and user experience. To evaluate it, we design a user study to compare different interactions for sketch generation. Second, we explore the combination of sketch input and vocal description to correct and fine-tune the search for 3D models in a database containing fine-grained variation. We analyse sketch and speech queries, identifying a way to incorporate both of them into our system's interaction loop. Third, in the context of the visual content production pipeline, we present a detailed study of visual metrics. We propose a novel method for detecting rendering-based artefacts in images. It exploits analogous deep learning algorithms used when extracting features from sketches

Utilizing a 3D game engine to develop a virtual design review system

A design review process is where information is exchanged between the designers and design reviewers to resolve any potential design related issues, and to ensure that the interests and goals of the owner are met. The effective execution of design review will minimize potential errors or conflicts, reduce the time for review, shorten the project life-cycle, allow for earlier occupancy, and ultimately translate into significant total project savings to the owner. However, the current methods of design review are still heavily relying on 2D paper-based format, sequential and lack central and integrated information base for efficient exchange and flow of information. There is thus a need for the use of a new medium that allow for 3D visualization of designs, collaboration among designers and design reviewers, and early and easy access to design review information. This paper documents the innovative utilization of a 3D game engine, the Torque Game Engine as the underlying tool and enabling technology for a design review system, the Virtual Design Review System for architectural designs. Two major elements are incorporated; 1) a 3D game engine as the driving tool for the development and implementation of design review processes, and 2) a virtual environment as the medium for design review, where visualization of design and design review information is based on sound principles of GUI design. The development of the VDRS involves two major phases; firstly, the creation of the assets and the assembly of the virtual environment, and secondly, the modification of existing functions or introducing new functionality through programming of the 3D game engine in order to support design review in a virtual environment. The features that are included in the VDRS are support for database, real-time collaboration across network, viewing and navigation modes, 3D object manipulation, parametric input, GUI, and organization for 3D objects

A virtual environment for the design and simulated construction of prefabricated buildings

The construction industry has acknowledged that its current working practices are in need of substantial improvements in quality and efficiency and has identified that computer modelling techniques and the use of prefabricated components can help reduce times, costs, and minimise defects and problems of on-site construction. This paper describes a virtual environment to support the design and construction processes of buildings from prefabricated components and the simulation of their construction sequence according to a project schedule. The design environment can import a library of 3-D models of prefabricated modules that can be used to interactively design a building. Using Microsoft Project, the construction schedule of the designed building can be altered, with this information feeding back to the construction simulation environment. Within this environment the order of construction can be visualised using virtual machines. Novel aspects of the system are that it provides a single 3-D environment where the user can construct their design with minimal user interaction through automatic constraint recognition and view the real-time simulation of the construction process within the environment. This takes this area of research a step forward from other systems that only allow the planner to view the construction at certain stages, and do not provide an animated view of the construction process

HBIM for conservation: A new proposal for information modeling

Thanks to its capability of archiving and organizing all the information about a building, HBIM (Historical Building Information Modeling) is considered a promising resource for planned conservation of historical assets. However, its usage remains limited and scarcely adopted by the subjects in charge of conservation, mainly because of its rather complex 3D modeling requirements and a lack of shared regulatory references and guidelines as far as semantic data are concerned. In this study, we developed an HBIM methodology to support documentation, management, and planned conservation of historic buildings, with particular focus on non-geometric information: organized and coordinated storage and management of historical data, easy analysis and query, time management, flexibility, user-friendliness, and information sharing. The system is based on a standalone specific-designed database linked to the 3D model of the asset, built with BIM software, and it is highly adaptable to different assets. The database is accessible both with a developed desktop application, which acts as a plug-in for the BIM software, and through a web interface, implemented to ensure data sharing and easy usability by skilled and unskilled users. The paper describes in detail the implemented system, passing by semantic breaking down of the building, database design, as well as system architecture and capabilities. Two case studies, the Cathedral of Parma and Ducal Palace of Mantua (Italy), are then presented to show the results of the system's application