V-ANFIS for Dealing with Visual Uncertainty for Force Estimation in Robotic Surgery

Accurate and robust estimation of applied forces in Robotic-Assisted Minimally Invasive Surgery is a very challenging task. Many vision-based solutions attempt to estimate the force by measuring the surface deformation after contacting the surgical tool. However, visual uncertainty, due to tool occlusion, is a major concern and can highly affect the results' precision. In this paper, a novel design of an adaptive neuro-fuzzy inference strategy with a voting step (V-ANFIS) is used to accommodate with this loss of information. Experimental results show a significant accuracy improvement from 50% to 77% with respect to other proposals.Peer ReviewedPostprint (published version

On combining the facial movements of a talking head

We present work on Obie, an embodied conversational agent framework. An embodied conversational agent, or talking head, consists of three main components. The graphical part consists of a face model and a facial muscle model. Besides the graphical part, we have implemented an emotion model and a mapping from emotions to facial expressions. The animation part of the framework focuses on the combination of different facial movements temporally. In this paper we propose a scheme of combining facial movements on a 3D talking head

Towards retrieving force feedback in robotic-assisted surgery: a supervised neuro-recurrent-vision approach

Robotic-assisted minimally invasive surgeries have gained a lot of popularity over conventional procedures as they offer many benefits to both surgeons and patients. Nonetheless, they still suffer from some limitations that affect their outcome. One of them is the lack of force feedback which restricts the surgeon's sense of touch and might reduce precision during a procedure. To overcome this limitation, we propose a novel force estimation approach that combines a vision based solution with supervised learning to estimate the applied force and provide the surgeon with a suitable representation of it. The proposed solution starts with extracting the geometry of motion of the heart's surface by minimizing an energy functional to recover its 3D deformable structure. A deep network, based on a LSTM-RNN architecture, is then used to learn the relationship between the extracted visual-geometric information and the applied force, and to find accurate mapping between the two. Our proposed force estimation solution avoids the drawbacks usually associated with force sensing devices, such as biocompatibility and integration issues. We evaluate our approach on phantom and realistic tissues in which we report an average root-mean square error of 0.02 N.Peer ReviewedPostprint (author's final draft

The development of a prototype intelligent user interface subsystem for NASA's scientific database systems

The National Space Science Data Center (NSSDC) has initiated an Intelligent Data Management (IDM) research effort which has as one of its components the development of an Intelligent User Interface (IUI).The intent of the latter is to develop a friendly and intelligent user interface service that is based on expert systems and natural language processing technologies. The purpose is to support the large number of potential scientific and engineering users presently having need of space and land related research and technical data but who have little or no experience in query languages or understanding of the information content or architecture of the databases involved. This technical memorandum presents prototype Intelligent User Interface Subsystem (IUIS) using the Crustal Dynamics Project Database as a test bed for the implementation of the CRUDDES (Crustal Dynamics Expert System). The knowledge base has more than 200 rules and represents a single application view and the architectural view. Operational performance using CRUDDES has allowed nondatabase users to obtain useful information from the database previously accessible only to an expert database user or the database designer

RECREATING AND SIMULATING DIGITAL COSTUMES FROM A STAGE PRODUCTION OF \u3ci\u3eMEDEA\u3c/i\u3e

This thesis investigates a technique to effectively construct and simulate costumes from a stage production Medea, in a dynamic cloth simulation application like Maya\u27s nDynamics. This was done by using data collected from real-world fabric tests and costume construction in the theatre\u27s costume studio. Fabric tests were conducted and recorded, by testing costume fabrics for drape and behavior with two collision objects. These tests were recreated digitally in Maya to derive appropriate parameters for the digital fabric, by comparing with the original reference. Basic mannequin models were created using the actors\u27 measurements and skeleton-rigged to enable animation. The costumes were then modeled and constrained according to the construction process observed in the costume studio to achieve the same style and stitch as the real costumes. Scenes selected and recorded from Medea were used as reference to animate the actors\u27 models. The costumes were assigned the parameters derived from the fabric tests to produce the simulations. Finally, the scenes were lit and rendered out to obtain the final videos which were compared to the original recordings to ascertain the accuracy of simulation. By obtaining and refining simulation parameters from simple fabric collision tests, and modeling the digital costumes following the procedures derived from real-life costume construction, realistic costume simulation was achieved

Stereoscopic Sketchpad: 3D Digital Ink

--Context-- This project looked at the development of a stereoscopic 3D environment in which a user is able to draw freely in all three dimensions. The main focus was on the storage and manipulation of the ‘digital ink’ with which the user draws. For a drawing and sketching package to be effective it must not only have an easy to use user interface, it must be able to handle all input data quickly and efficiently so that the user is able to focus fully on their drawing. --Background-- When it comes to sketching in three dimensions the majority of applications currently available rely on vector based drawing methods. This is primarily because the applications are designed to take a users two dimensional input and transform this into a three dimensional model. Having the sketch represented as vectors makes it simpler for the program to act upon its geometry and thus convert it to a model. There are a number of methods to achieve this aim including Gesture Based Modelling, Reconstruction and Blobby Inflation. Other vector based applications focus on the creation of curves allowing the user to draw within or on existing 3D models. They also allow the user to create wire frame type models. These stroke based applications bring the user closer to traditional sketching rather than the more structured modelling methods detailed. While at present the field is inundated with vector based applications mainly focused upon sketch-based modelling there are significantly less voxel based applications. The majority of these applications focus on the deformation and sculpting of voxmaps, almost the opposite of drawing and sketching, and the creation of three dimensional voxmaps from standard two dimensional pixmaps. How to actually sketch freely within a scene represented by a voxmap has rarely been explored. This comes as a surprise when so many of the standard 2D drawing programs in use today are pixel based. --Method-- As part of this project a simple three dimensional drawing program was designed and implemented using C and C++. This tool is known as Sketch3D and was created using a Model View Controller (MVC) architecture. Due to the modular nature of Sketch3Ds system architecture it is possible to plug a range of different data structures into the program to represent the ink in a variety of ways. A series of data structures have been implemented and were tested for efficiency. These structures were a simple list, a 3D array, and an octree. They have been tested for: the time it takes to insert or remove points from the structure; how easy it is to manipulate points once they are stored; and also how the number of points stored effects the draw and rendering times. One of the key issues brought up by this project was devising a means by which a user is able to draw in three dimensions while using only two dimensional input devices. The method settled upon and implemented involves using the mouse or a digital pen to sketch as one would in a standard 2D drawing package but also linking the up and down keyboard keys to the current depth. This allows the user to move in and out of the scene as they draw. A couple of user interface tools were also developed to assist the user. A 3D cursor was implemented and also a toggle, which when on, highlights all of the points intersecting the depth plane on which the cursor currently resides. These tools allow the user to see exactly where they are drawing in relation to previously drawn lines. --Results-- The tests conducted on the data structures clearly revealed that the octree was the most effective data structure. While not the most efficient in every area, it manages to avoid the major pitfalls of the other structures. The list was extremely quick to render and draw to the screen but suffered severely when it comes to finding and manipulating points already stored. In contrast the three dimensional array was able to erase or manipulate points effectively while the draw time rendered the structure effectively useless, taking huge amounts of time to draw each frame. The focus of this research was on how a 3D sketching package would go about storing and accessing the digital ink. This is just a basis for further research in this area and many issues touched upon in this paper will require a more in depth analysis. The primary area of this future research would be the creation of an effective user interface and the introduction of regular sketching package features such as the saving and loading of images

Guided Medical Data Segmentation Using Structure-Aligned Planar Contours

Segmentation of 3D/4D biological images is a critical step for a wide range of applications such as treatment planning, quantitative analysis, virtual simulations, and rendering visualizations. Automatic segmentation methods are becoming more reliable, but many experts still rely on manual intervention which makes segmentation a time and resource intensive bottleneck. Marking boundary contours in 3D images can be difficult when images are often noisy or the delineation of biological tissue is unclear. Non-parallel contours can be more accurate and reduce the amount of marking necessary, but require extra effort to ensure boundary consistency and maintain spatial orientation. This dissertation focuses three problems that pertain to drawing non-parallel contour networks and generating a segmentation surface from those networks. First a guided structure-aligned segmentation system is detailed that utilizes prior structure knowledge from past segmentations of similar data. It employs a contouring protocol to aid in navigating the volume data and support using arbitrarily-oriented contouring planes placed to capture or follow the global structure shape. A user study is provided to test how well novices perform segmentation using this system. The following two problems then aim to improve different aspects of this system. A new deformation approach to reconstruction is discussed which deforms previous segmentation meshes to fit protocol drawn contours from new data instances in order to obtain accurate segmentations that have the correct topology and general shape and preserves fine details. The focus is on the problem of finding a correspondence between a mesh and a set of contours describing a similar shape. And finally, a new robust algorithm that resolves inconsistencies in contour networks is detailed. Inconsistent contours are faster and less demanding to draw, and they allow the segmenter to focus on drawing boundaries and not maintaining consistency. However, inconsistency is detrimental to most reconstruction algorithms, so the network must be fixed as a post process after drawing