A virtual environment for the modelling, simulation and manufacturing of orthopaedic devices

This thesis was submitted for the degree of Doctor of Philosophy and awarded by Brunel University.The objective of this work is to investigate whether the game physics based modelling is accurate enough to be used in modelling the motion of the human body, in particular musculoskeletal motion. Hitherto, the implementation of game physics in the medical field focused only on anatomical representation for education and training purposes. Introducing gaming platforms and physics engines into orthopaedics applications will help to overcome several difficulties encountered in the modelling of articular joints. Implementing a physics engine (PhysX), which is mainly designed for video games, handles intensive computations in optimized ways at an interactive speed. In this study, the capabilities of the physics engine (PhysX) and gaming platform for modelling and simulating articular joints are evaluated. First, a preliminary validation is carried out for mechanical systems with analytical solutions, before constructing the musculoskeletal model to evaluate the consistency of gaming platforms. The developed musculoskeletal model deals with the human joint as an unconstrained system with 6 DOF which is not available with other joint modeller. The model articulation is driven by contact surfaces and the stiffness of surrounding tissues. A number of contributions, such as contact modelling and muscle wrapping, have been made in this research to overcome some existing challenges in joint modelling. Using muscle segmentation, the proposed technique effectively handles the problem of muscle wrapping, a major concern for many; thus the shortest path and line of action are no longer problematic. Collision behaviour has also shown a stable response for colliding as well as resting objects, provided that it is based on the principles of surface properties and the conservation of linear and angular momentums. The precision of collision detection and response are within an acceptable tolerance controllable by varying the mesh density. An image based analysis system is developed in this thesis, mainly in order to validate the proposed physics based modelling solution. This minimally invasive method is based on the analysis of marker positions located at bony positions with minimal skin movement. The image based system overcomes several challenges associated with the currently existing methods, such as inaccuracy, complication, impracticability and cost. The analysis part of this research has considered the elbow joint as a case study to investigate and validate the proposed physics based model. Beside the interactive 3D simulation, the obtained results are validated by comparing them with the image based system developed within the current research to investigate joint kinematics and laxity and also with published material, MJM and results from experiments performed at the Brunel Orthopaedic Research and Learning Centre. The proposed modelling shows the advantageous speed, reliability and flexibility of the proposed model. It is shown that the gaming platform and physics engine provide a viable solution to human musculoskeletal modelling. Finally, this thesis considers an extended implementation of the proposed platform for testing and assessing the design of custom-made implants, to enhance joint performance. The developed simulation software is expected to give indicative results as well as testing different types of prosthetic implant. Design parameterization and sensitivity analysis for geometrical features are discussed. Thus, an integrated environment is proposed to link the real-time simulation software with a manufacturing environment so as to assist the production of patient specific implants by rapid manufacturing

A constraint manager to support virtual maintainability

Virtual prototyping tools have already captivated the industry's interest as viable design tool. One of the key challenges for the research community is to extend the capabilities of Virtual Reality technology beyond its current scope of ergonomics and design reviews. The research presented in this paper is part of a larger research programme that aims to perform maintainability assessment on virtual prototypes. This paper discusses the design and implementation of a geometric constraint manager that has been designed to support physical realism and interactive assembly and disassembly tasks within virtual environments. The key techniques employed by the constraint manager are direct interaction, automatic constraint recognition, constraint satisfaction and constrained motion. Various optimization techniques have been implemented to achieve real-time interaction with large industrial models

A Multi-Sensorial Hybrid Control for Robotic Manipulation in Human-Robot Workspaces

Autonomous manipulation in semi-structured environments where human operators can interact is an increasingly common task in robotic applications. This paper describes an intelligent multi-sensorial approach that solves this issue by providing a multi-robotic platform with a high degree of autonomy and the capability to perform complex tasks. The proposed sensorial system is composed of a hybrid visual servo control to efficiently guide the robot towards the object to be manipulated, an inertial motion capture system and an indoor localization system to avoid possible collisions between human operators and robots working in the same workspace, and a tactile sensor algorithm to correctly manipulate the object. The proposed controller employs the whole multi-sensorial system and combines the measurements of each one of the used sensors during two different phases considered in the robot task: a first phase where the robot approaches the object to be grasped, and a second phase of manipulation of the object. In both phases, the unexpected presence of humans is taken into account. This paper also presents the successful results obtained in several experimental setups which verify the validity of the proposed approach

A novel musculoskeletal joint modelling for orthopaedic applications

This thesis was submitted for the degree of Docter of Philosophy and awarded by Brunel University.The objective of the work carried out in this thesis was to develop analytical and computational tools to model and investigate musculoskeletal human joints. It was recognised that the FEA was used by many researchers in modelling human musculoskeletal motion, loading and stresses. However the continuum mechanics played only a minor role in determining the articular joint motion, and its value was questionable. This is firstly due to the computational cost and secondly due to its impracticality for this application. On the other hand, there isn’t any suitable software for precise articular joint motion analysis to deal with the local joint stresses or non standard joints. The main requirement in orthopaedics field is to develop a modeller software (and its associated theories) to model anatomic joint as it is, without any simplification with respect to joint surface morphology and material properties of surrounding tissues. So that the proposed modeller can be used for evaluating and diagnosing different joint abnormalities but furthermore form the basis for performing implant insertion and analysis of the artificial joints. The work which is presented in this thesis is a new frame work and has been developed for human anatomic joint analysis which describes the joint in terms of its surface geometry and surrounding musculoskeletal tissues. In achieving such a framework several contributions were made to the 6DOF linear and nonlinear joint modelling, the mathematical definition of joint stiffness, tissue path finding and wrapping and the contact with collision analysis. In 6DOF linear joint modelling, the contribution is the development of joint stiffness and damping matrices. This modelling approach is suitable for the linear range of tissue stiffness and damping properties. This is the first of its kind and it gives a firm analytical basis for investigating joints with surrounding tissue and the cartilage. The 6DOF nonlinear joint modelling is a new scheme which is described for modelling the motion of multi bodies joined by non-linear stiffness and contact elements. The proposed method requires no matrix assembly for the stiffness and damping elements or mass elements. The novelty in the nonlinear modelling, relates to the overall algorithmic approach and handling local non-linearity by procedural means. The mathematical definition of joint stiffness is also a new proposal which is based on the mathematical definition of stiffness between two bodies. Based on the joint stiffness matrix properties, number of joint stiffness invariants was obtained analytically such as the centre of stiffness, the principal translational stiffnesses, and the principal rotational stiffnesses. In corresponding to these principal stiffnesses, their principal axes have been also obtained. Altogether, a joint is assessed by six principal axes and six principal stiffnesses and its centre of stiffness. These formulations are new and show that a joint can be described in terms of inherent stiffness properties. It is expected that these will be better in characterising a joint in comparison to laxity based characterisation. The development of tissue path finding and wrapping algorithms are also introduced as new approaches. The musculoskeletal tissue wrapping involves calculating the shortest distance between two points on a meshed surface. A new heuristic algorithm was proposed. The heuristic is based on minimising the accumulative divergence from the straight line between two points on the surface and the direction of travel on the surface (i.e. bone). In contact and collision based development, the novel algorithm has been proposed that detects possible colliding points on the motion trajectory by redefining the distance as a two dimensional measure along the velocity approach vector and perpendicular to this vector. The perpendicular distance determines if there are potentially colliding points, and the distance along the velocity determines how close they are. The closest pair among the potentially colliding points gives the “time to collision”. The algorithm can eliminate the “fly pass” situation where very close points may not collide because of the direction of their relative velocity. All these developed algorithms and modelling theories, have been encompassed in the developed prototype software in order to simulate the anatomic joint articulations through modelling formulations developed. The software platform provides a capability for analysing joints as 6DOF joints based on anatomic joint surfaces. The software is highly interactive and driven by well structured database, designed to be highly flexible for the future developments. Particularly, two case studies are carried out in this thesis in order to generate results relating to all the proposed elements of the study. The results obtained from the case studies show good agreement with previously published results or model based results obtained from Lifemod software, whenever comparison was possible. In some cases the comparison was not possible because there were no equivalent results; the results were supported by other indicators. The modelling based results were also supported by experiments performed in the Brunel Orthopaedic Research and Learning Centre

Designing 3D scenarios and interaction tasks for immersive environments

In the world of today, immersive reality such as virtual and mixed reality, is one of the most attractive research fields. Virtual Reality, also called VR, has a huge potential to be used in in scientific and educational domains by providing users with real-time interaction or manipulation. The key concept in immersive technologies to provide a high level of immersive sensation to the user, which is one of the main challenges in this field. Wearable technologies play a key role to enhance the immersive sensation and the degree of embodiment in virtual and mixed reality interaction tasks. This project report presents an application study where the user interacts with virtual objects, such as grabbing objects, open or close doors and drawers while wearing a sensory cyberglove developed in our lab (Cyberglove-HT). Furthermore, it presents the development of a methodology that provides inertial measurement unit(IMU)-based gesture recognition. The interaction tasks and 3D immersive scenarios were designed in Unity 3D. Additionally, we developed an inertial sensor-based gesture recognition by employing an Long short-term memory (LSTM) network. In order to distinguish the effect of wearable technologies in the user experience in immersive environments, we made an experimental study comparing the Cyberglove-HT to standard VR controllers (HTC Vive Controller). The quantitive and subjective results indicate that we were able to enhance the immersive sensation and self embodiment with the Cyberglove-HT. A publication resulted from this work [1] which has been developed in the framework of the R&D project Human Tracking and Perception in Dynamic Immersive Rooms (HTPDI

Collision Detection and Merging of Deformable B-Spline Surfaces in Virtual Reality Environment

This thesis presents a computational framework for representing, manipulating and merging rigid and deformable freeform objects in virtual reality (VR) environment. The core algorithms for collision detection, merging, and physics-based modeling used within this framework assume that all 3D deformable objects are B-spline surfaces. The interactive design tool can be represented as a B-spline surface, an implicit surface or a point, to allow the user a variety of rigid or deformable tools. The collision detection system utilizes the fact that the blending matrices used to discretize the B-spline surface are independent of the position of the control points and, therefore, can be pre-calculated. Complex B-spline surfaces can be generated by merging various B-spline surface patches using the B-spline surface patches merging algorithm presented in this thesis. Finally, the physics-based modeling system uses the mass-spring representation to determine the deformation and the reaction force values provided to the user. This helps to simulate realistic material behaviour of the model and assist the user in validating the design before performing extensive product detailing or finite element analysis using commercially available CAD software. The novelty of the proposed method stems from the pre-calculated blending matrices used to generate the points for graphical rendering, collision detection, merging of B-spline patches, and nodes for the mass spring system. This approach reduces computational time by avoiding the need to solve complex equations for blending functions of B-splines and perform the inversion of large matrices. This alternative approach to the mechanical concept design will also help to do away with the need to build prototypes for conceptualization and preliminary validation of the idea thereby reducing the time and cost of concept design phase and the wastage of resources

Real-time Immersive human-computer interaction based on tracking and recognition of dynamic hand gestures

With fast developing and ever growing use of computer based technologies, human-computer interaction (HCI) plays an increasingly pivotal role. In virtual reality (VR), HCI technologies provide not only a better understanding of three-dimensional shapes and spaces, but also sensory immersion and physical interaction. With the hand based HCI being a key HCI modality for object manipulation and gesture based communication, challenges are presented to provide users a natural, intuitive, effortless, precise, and real-time method for HCI based on dynamic hand gestures, due to the complexity of hand postures formed by multiple joints with high degrees-of-freedom, the speed of hand movements with highly variable trajectories and rapid direction changes, and the precision required for interaction between hands and objects in the virtual world. Presented in this thesis is the design and development of a novel real-time HCI system based on a unique combination of a pair of data gloves based on fibre-optic curvature sensors to acquire finger joint angles, a hybrid tracking system based on inertia and ultrasound to capture hand position and orientation, and a stereoscopic display system to provide an immersive visual feedback. The potential and effectiveness of the proposed system is demonstrated through a number of applications, namely, hand gesture based virtual object manipulation and visualisation, hand gesture based direct sign writing, and hand gesture based finger spelling. For virtual object manipulation and visualisation, the system is shown to allow a user to select, translate, rotate, scale, release and visualise virtual objects (presented using graphics and volume data) in three-dimensional space using natural hand gestures in real-time. For direct sign writing, the system is shown to be able to display immediately the corresponding SignWriting symbols signed by a user using three different signing sequences and a range of complex hand gestures, which consist of various combinations of hand postures (with each finger open, half-bent, closed, adduction and abduction), eight hand orientations in horizontal/vertical plans, three palm facing directions, and various hand movements (which can have eight directions in horizontal/vertical plans, and can be repetitive, straight/curve, clockwise/anti-clockwise). The development includes a special visual interface to give not only a stereoscopic view of hand gestures and movements, but also a structured visual feedback for each stage of the signing sequence. An excellent basis is therefore formed to develop a full HCI based on all human gestures by integrating the proposed system with facial expression and body posture recognition methods. Furthermore, for finger spelling, the system is shown to be able to recognise five vowels signed by two hands using the British Sign Language in real-time

Analysis and synthesis of bipedal humanoid movement : a physical simulation approach

textAdvances in graphics and robotics have increased the importance of tools for synthesizing humanoid movements to control animated characters and physical robots. There is also an increasing need for analyzing human movements for clinical diagnosis and rehabilitation. Existing tools can be expensive, inefficient, or difficult to use. Using simulated physics and motion capture to develop an interactive virtual reality environment, we capture natural human movements in response to controlled stimuli. This research then applies insights into the mathematics underlying physics simulation to adapt the physics solver to support many important tasks involved in analyzing and synthesizing humanoid movement. These tasks include fitting an articulated physical model to motion capture data, modifying the model pose to achieve a desired configuration (inverse kinematics), inferring internal torques consistent with changing pose data (inverse dynamics), and transferring a movement from one model to another model (retargeting). The result is a powerful and intuitive process for analyzing and synthesizing movement in a single unified framework.Computer Science