Recent Developments and Future Challenges in Medical Mixed Reality

As AR technology matures, we have seen many applicationsemerge in entertainment, education and training. However, the useof AR is not yet common in medical practice, despite the great po-tential of this technology to help not only learning and training inmedicine, but also in assisting diagnosis and surgical guidance. Inthis paper, we present recent trends in the use of AR across all med-ical specialties and identify challenges that must be overcome tonarrow the gap between academic research and practical use of ARin medicine. A database of 1403 relevant research papers publishedover the last two decades has been reviewed by using a novel re-search trend analysis method based on text mining algorithm. Wesemantically identified 10 topics including varies of technologiesand applications based on the non-biased and in-personal cluster-ing results from the Latent Dirichlet Allocatio (LDA) model andanalysed the trend of each topic from 1995 to 2015. The statisticresults reveal a taxonomy that can best describes the developmentof the medical AR research during the two decades. And the trendanalysis provide a higher level of view of how the taxonomy haschanged and where the focus will goes. Finally, based on the valu-able results, we provide a insightful discussion to the current limi-tations, challenges and future directions in the field. Our objectiveis to aid researchers to focus on the application areas in medicalAR that are most needed, as well as providing medical practitioners with latest technology advancements

Mobile Augmented Reality: User Interfaces, Frameworks, and Intelligence

Mobile Augmented Reality (MAR) integrates computer-generated virtual objects with physical environments for mobile devices. MAR systems enable users to interact with MAR devices, such as smartphones and head-worn wearables, and perform seamless transitions from the physical world to a mixed world with digital entities. These MAR systems support user experiences using MAR devices to provide universal access to digital content. Over the past 20 years, several MAR systems have been developed, however, the studies and design of MAR frameworks have not yet been systematically reviewed from the perspective of user-centric design. This article presents the first effort of surveying existing MAR frameworks (count: 37) and further discuss the latest studies on MAR through a top-down approach: (1) MAR applications; (2) MAR visualisation techniques adaptive to user mobility and contexts; (3) systematic evaluation of MAR frameworks, including supported platforms and corresponding features such as tracking, feature extraction, and sensing capabilities; and (4) underlying machine learning approaches supporting intelligent operations within MAR systems. Finally, we summarise the development of emerging research fields and the current state-of-the-art, and discuss the important open challenges and possible theoretical and technical directions. This survey aims to benefit both researchers and MAR system developers alike.Peer reviewe

Augmented reality (AR) for surgical robotic and autonomous systems: State of the art, challenges, and solutions

Despite the substantial progress achieved in the development and integration of augmented reality (AR) in surgical robotic and autonomous systems (RAS), the center of focus in most devices remains on improving end-effector dexterity and precision, as well as improved access to minimally invasive surgeries. This paper aims to provide a systematic review of different types of state-of-the-art surgical robotic platforms while identifying areas for technological improvement. We associate specific control features, such as haptic feedback, sensory stimuli, and human-robot collaboration, with AR technology to perform complex surgical interventions for increased user perception of the augmented world. Current researchers in the field have, for long, faced innumerable issues with low accuracy in tool placement around complex trajectories, pose estimation, and difficulty in depth perception during two-dimensional medical imaging. A number of robots described in this review, such as Novarad and SpineAssist, are analyzed in terms of their hardware features, computer vision systems (such as deep learning algorithms), and the clinical relevance of the literature. We attempt to outline the shortcomings in current optimization algorithms for surgical robots (such as YOLO and LTSM) whilst providing mitigating solutions to internal tool-to-organ collision detection and image reconstruction. The accuracy of results in robot end-effector collisions and reduced occlusion remain promising within the scope of our research, validating the propositions made for the surgical clearance of ever-expanding AR technology in the future

A Survey on Augmented Reality Challenges and Tracking

This survey paper presents a classification of different challenges and tracking techniques in the field of augmented reality. The challenges in augmented reality are categorized into performance challenges, alignment challenges, interaction challenges, mobility/portability challenges and visualization challenges. Augmented reality tracking techniques are mainly divided into sensor-based tracking, visionbased tracking and hybrid tracking. The sensor-based tracking is further divided into optical tracking, magnetic tracking, acoustic tracking, inertial tracking or any combination of these to form hybrid sensors tracking. Similarly, the vision-based tracking is divided into marker-based tracking and markerless tracking. Each tracking technique has its advantages and limitations. Hybrid tracking provides a robust and accurate tracking but it involves financial and tehnical difficulties

Towards markerless orthopaedic navigation with intuitive Optical See-through Head-mounted displays

The potential of image-guided orthopaedic navigation to improve surgical outcomes has been well-recognised during the last two decades. According to the tracked pose of target bone, the anatomical information and preoperative plans are updated and displayed to surgeons, so that they can follow the guidance to reach the goal with higher accuracy, efficiency and reproducibility. Despite their success, current orthopaedic navigation systems have two main limitations: for target tracking, artificial markers have to be drilled into the bone and calibrated manually to the bone, which introduces the risk of additional harm to patients and increases operating complexity; for guidance visualisation, surgeons have to shift their attention from the patient to an external 2D monitor, which is disruptive and can be mentally stressful. Motivated by these limitations, this thesis explores the development of an intuitive, compact and reliable navigation system for orthopaedic surgery. To this end, conventional marker-based tracking is replaced by a novel markerless tracking algorithm, and the 2D display is replaced by a 3D holographic Optical see-through (OST) Head-mounted display (HMD) precisely calibrated to a user's perspective. Our markerless tracking, facilitated by a commercial RGBD camera, is achieved through deep learning-based bone segmentation followed by real-time pose registration. For robust segmentation, a new network is designed and efficiently augmented by a synthetic dataset. Our segmentation network outperforms the state-of-the-art regarding occlusion-robustness, device-agnostic behaviour, and target generalisability. For reliable pose registration, a novel Bounded Iterative Closest Point (BICP) workflow is proposed. The improved markerless tracking can achieve a clinically acceptable error of 0.95 deg and 2.17 mm according to a phantom test. OST displays allow ubiquitous enrichment of perceived real world with contextually blended virtual aids through semi-transparent glasses. They have been recognised as a suitable visual tool for surgical assistance, since they do not hinder the surgeon's natural eyesight and require no attention shift or perspective conversion. The OST calibration is crucial to ensure locational-coherent surgical guidance. Current calibration methods are either human error-prone or hardly applicable to commercial devices. To this end, we propose an offline camera-based calibration method that is highly accurate yet easy to implement in commercial products, and an online alignment-based refinement that is user-centric and robust against user error. The proposed methods are proven to be superior to other similar State-of- the-art (SOTA)s regarding calibration convenience and display accuracy. Motivated by the ambition to develop the world's first markerless OST navigation system, we integrated the developed markerless tracking and calibration scheme into a complete navigation workflow designed for femur drilling tasks during knee replacement surgery. We verify the usability of our designed OST system with an experienced orthopaedic surgeon by a cadaver study. Our test validates the potential of the proposed markerless navigation system for surgical assistance, although further improvement is required for clinical acceptance.Open Acces

Kalaia: branding through emerging technologies

Companies have existed for thousands of years all over the world, however, brands are the personality of a company, and as people, a broad range of personalities can be found, just like brands. The choice and use of the appropriate brand language according to the business’s vision, mission and value create the correct perception to its targeted customers. The name, the logo, the colours, the imagery chosen is the face of the brand. This thesis presents how and why to choose appropriate brand language, bringing to it added value, such as understanding why purple is more than mixing red and blue, or why choosing a particular shaped letter is more than a angled line. Just as brands change, times change as well; it is important to take into consideration the era we live in and the eras your brand will live in, this being longer than its founders. With this is mind, it was important to understand what the technological trends of the decade are, and what to expect as the billboards of tomorrow. Augmented reality was a trend which caught my attention and decided to investigate and implement while projecting creation of the brand. Considering the objective of creating a fashion brand in a technological era, it was important to conduct case studies of fashion brands, understand brand language and how augmented reality is implemented in their advertisement. Kalaia, a swimwear brand, I created with the development of this thesis is the ensemble of both theoretical and projection components form this course. Kalaia, selling products made in Portugal, intends to sell more than swimsuits but a lifestyle, for daring, irreverent and adventurous women.Empresas existem há centenas de anos, no entanto, as marcas são a personalidade de uma empresa e, como pessoas, uma grande variedade de personalidades pode ser encontrada em diferentes marcas. A escolha e o uso da linguagem visual apropriada, de acordo com a visão, missão e valores da empresa criam a perceção correta paro o público-alvo pretendido. O nome, o logotipo, as cores, as imagens escolhidas são a personalidade que encontramos nas marcas. Esta tese apresenta escolher a linguagem apropriada da marca para lhe acrescentar valor, tal como perceber que o roxo é mais do que a mistura de vermelho e azul ou que uma letra é mais que linhas angulares. Assim como as marcas mudam, os tempos também e é importante levar em consideração a época em que vivemos e as épocas em que uma marca ir permanecer. Com isso em mente, era importante entender quais são as tendências tecnológicas da década e o que esperar da publicidade de amanhã. A realidade aumentada foi uma tendência que me chamou à atenção e decidi investigar e implementar para a projeção e criação da marca. Considerando o objetivo de criar uma marca de moda durante uma era tecnológica, era importante realizar estudos de caso de marcas de moda, e compreender a linguagem visual utilizadas assim como a realidade aumentada implementada nas suas publicidades. Kalaia, uma marca de moda de praia que criei com o desenvolvimento desta tese, é o conjunto de componentes teóricos e práticos deste curso. A Kalaia, que vende produtos fabricados em Portugal, pretende vender mais do que fatos de banho, mas um estilo de vida para mulheres ousadas, e irreverentes

Cultural Heritage in Marker-Less Augmented Reality: A Survey

Augmented Reality (AR) is considered as one of the most significant technologies in the field of computer graphics and is utilized for many applications. In this paper, we have presented a comprehensive survey for cultural heritage using Augmented Reality systems. This survey describes the main objectives and characteristics of Marker-less Augmented Reality Systems through presenting up-to-date research results in this area. We describe the marker-less technologies in the area of AR, indoor marker-less AR, outdoor marker-less AR, real-time solutions to the tracking problem, real-time registration, cultural heritage in AR, 3D remonstration techniques, as well as presenting the problems in each research