Optical techniques for 3D surface reconstruction in computer-assisted laparoscopic surgery

One of the main challenges for computer-assisted surgery (CAS) is to determine the intra-opera- tive morphology and motion of soft-tissues. This information is prerequisite to the registration of multi-modal patient-specific data for enhancing the surgeon’s navigation capabilites by observ- ing beyond exposed tissue surfaces and for providing intelligent control of robotic-assisted in- struments. In minimally invasive surgery (MIS), optical techniques are an increasingly attractive approach for in vivo 3D reconstruction of the soft-tissue surface geometry. This paper reviews the state-of-the-art methods for optical intra-operative 3D reconstruction in laparoscopic surgery and discusses the technical challenges and future perspectives towards clinical translation. With the recent paradigm shift of surgical practice towards MIS and new developments in 3D opti- cal imaging, this is a timely discussion about technologies that could facilitate complex CAS procedures in dynamic and deformable anatomical regions

Surgical robotics beyond enhanced dexterity instrumentation: a survey of machine learning techniques and their role in intelligent and autonomous surgical actions

PURPOSE: Advances in technology and computing play an increasingly important role in the evolution of modern surgical techniques and paradigms. This article reviews the current role of machine learning (ML) techniques in the context of surgery with a focus on surgical robotics (SR). Also, we provide a perspective on the future possibilities for enhancing the effectiveness of procedures by integrating ML in the operating room. METHODS: The review is focused on ML techniques directly applied to surgery, surgical robotics, surgical training and assessment. The widespread use of ML methods in diagnosis and medical image computing is beyond the scope of the review. Searches were performed on PubMed and IEEE Explore using combinations of keywords: ML, surgery, robotics, surgical and medical robotics, skill learning, skill analysis and learning to perceive. RESULTS: Studies making use of ML methods in the context of surgery are increasingly being reported. In particular, there is an increasing interest in using ML for developing tools to understand and model surgical skill and competence or to extract surgical workflow. Many researchers begin to integrate this understanding into the control of recent surgical robots and devices. CONCLUSION: ML is an expanding field. It is popular as it allows efficient processing of vast amounts of data for interpreting and real-time decision making. Already widely used in imaging and diagnosis, it is believed that ML will also play an important role in surgery and interventional treatments. In particular, ML could become a game changer into the conception of cognitive surgical robots. Such robots endowed with cognitive skills would assist the surgical team also on a cognitive level, such as possibly lowering the mental load of the team. For example, ML could help extracting surgical skill, learned through demonstration by human experts, and could transfer this to robotic skills. Such intelligent surgical assistance would significantly surpass the state of the art in surgical robotics. Current devices possess no intelligence whatsoever and are merely advanced and expensive instruments

XRSpotlight: Example-based Programming of XR Interactions using a Rule-based Approach

Research on enabling novice AR/VR developers has emphasized the need to lower the technical barriers to entry. This is often achieved by providing new authoring tools that provide simpler means to implement XR interactions through abstraction. However, novices are then bound by the ceiling of each tool and may not form the correct mental model of how interactions are implemented. We present XRSpotlight, a system that supports novices by curating a list of the XR interactions defined in a Unity scene and presenting them as rules in natural language. Our approach is based on a model abstraction that unifies existing XR toolkit implementations. Using our model, XRSpotlight can find incomplete specifications of interactions, suggest similar interactions, and copy-paste interactions from examples using different toolkits. We assess the validity of our model with professional VR developers and demonstrate that XRSpotlight helps novices understand how XR interactions are implemented in examples and apply this knowledge in their projects

SURVEY TOOL ALIGNMENT FOR REAL-TIME MIXED REALITY INFORMATION MODEL INTERACTION IN HERITAGE RECORDING

While heritage documentation today is largely based on three-dimensional models as key element, the easy access to tools for creating such models as well as quick and easier to use workflows lead to a stark increase in data creation while being followed by less structured and standardized methodology. Increasingly information models are applied to structure all available data regarding an object and reference it within a digital twin of the physical object. Nevertheless, various approaches reviewed all show a process of information model creating as post-processing task, undoubtedly leading to parallel systems of captured data, and disseminated data.The following research proposes to encounter the mentioned issues by developing a strongly interwoven integrative building documentation workflow with including GNSS-RTK based survey and augmented reality tools, to start compiling coordinate-based information models right on site during the first acquirement of data. As much data as possible should be included into this process, covering geometry, visual observations, and semantic information. Making data available right after inclusion into the information model should then allow to give real-time feedback on all data available (old and new) right on site or remotely within a web-platform, virtual reality interface or else. Detaching the work of curating and information model from a manual processing step, but rather being able to link information through their position right at the time of origin, would allow models to be created, enhanced, and expanded collaboratively further giving them a fourth dimension of time and collective knowledge.</p

IMPACT OF AUGMENTED DOCUMENTATION WORKFLOWS ON TRANSPORTATION AGENCY ASPHALT PAVING OPERATIONS: A QUALITATIVE AND QUANTITATIVE STUDY

e-Ticketing was an intriguing technology to many state transportation agencies (STAs) pre-COVID-19 pandemic but gained significant attention upon arrival of the pandemic due to the contactless nature of the technology. Research completed prior to the pandemic into e-ticketing for asphalt paving primarily identified qualitative benefits and concerns. However, minimal academic literature exists in the post-pandemic era discussing the additional e-ticketing benefits not previously captured which resulted in increased implementation by STAs. This research seeks to address this gap by gathering information from state members of the American Association of State Highway and Transportation Officials Committee on Construction (AASHTO COC), employees of the Kentucky Transportation Cabinet (KYTC), and members of the Kentucky Association of Highway Contractors (KAHC) and the Plantmix Asphalt Industry of Kentucky (PAIKY) regarding e-ticketing practices for asphalt paving in the post-pandemic era. The first portion of the study combines national and state-level survey responses to create a qualitative benefit-cost analysis for e-ticketing for asphalt paving operations based upon experience gained through emergency implementation during the pandemic. To reinforce the results of the qualitative analysis and increase the power of the study, the second portion of the research creates a quantitative benefit-cost analysis (BCA) from data collected from KYTC construction projects to compare the traditional weigh ticket collection process with e-ticketing processes for both project engineers and inspectors. The analysis shows a statistically significant time savings for field employees but not project management personnel. The monetary analysis for Kentucky indicates that while e-ticketing does not result in significant time savings for all employees, there is substantial monetary benefit worthy of STAs adopting e-ticketing as policy. Cluster analysis was completed for both project engineers and inspectors to group project types where e-ticketing impact was maximized, and a decision matrix was created to aid agencies in creating implementation plans. The primary contributions to the body of knowledge include an improved BCA methodology utilizing a modified action research approach applied to a new domain of electronic bulk material tickets and an informed quantitative valuation framework useful for STAs as a proof-of-concept to champion and effectively implement e-ticketing and by transportation-focused researchers for evaluating emerging technological applications

Implementing virtual reality-based design review in the construction industry

Building Information Modeling (BIM)-enabled digitalization has been innovating the entire workflow and information management in the Architecture, Engineering, and Construction (AEC) industry. Emerging visualization technologies such as Virtual Reality (VR) have extended BIM-based 3D visualization to an immersive and intuitive virtual prototyping experience. VR applications have shown their potential for making construction design review tasks more effective and collaborative, thus attracting growing research attention and efforts. However, despite the increasing technological capabilities, the adoption of VR applications has still been lagging in the AEC industry, leaving the theoretical potential unrealized. One major reason for the gap between the state-of-the-art and state-of-practice is the lack of validation on VR performance. Therefore, the first research question in this dissertation addressed this issue by measuring user performance when undertaking four design review tasks using VR-based or desktop-based devices. The experiment controlled more testbed-related variables, such as functions and navigation methods, than existing VR performance validation research to reveal the contribution exclusively from VR technology. Results show that VR users can detect significantly more design errors and commit significantly fewer mistakes in scheduling installation sequence when using VR as compared to desktop-based interactions. Experiments in the first research question revealed the lack of approaches to visualize occluded objects in 3D models. In order to facilitate a comprehensive design review, the second research question proposed a semi-automatic occlusion detection framework for building information models. It converts objects in a 3D model to point clouds and overlaps them with a virtual scan of the model to identify occlusion. A ready-to-implement VR application does not guarantee industry adoption because the development cost for VR can be demanding. The last part of this dissertation proposed a Generic Extended Reality (GenXR) model that supports all XR development in the AEC industry. The author compared BIM-to-XR durations in two case studies and six XR prototype development. The result shows that the GenXR model could save over 65% of model transfer time for XR development.Civil, Architectural, and Environmental Engineerin

Situation Interpretation for Knowledge- and Model Based Laparoscopic Surgery

To manage the influx of information into surgical practice, new man-machine interaction methods are necessary to prevent information overflow. This work presents an approach to automatically segment surgeries into phases and select the most appropriate pieces of information for the current situation. This way, assistance systems can adopt themselves to the needs of the surgeon and not the other way around

Value Creation with Extended Reality Technologies - A Methodological Approach for Holistic Deployments

Mit zunehmender Rechenkapazität und Übertragungsleistung von Informationstechnologien wächst die Anzahl möglicher Anwendungs-szenarien für Extended Reality (XR)-Technologien in Unternehmen. XR-Technologien sind Hardwaresysteme, Softwaretools und Methoden zur Erstellung von Inhalten, um Virtual Reality, Augmented Reality und Mixed Reality zu erzeugen. Mit der Möglichkeit, Nutzern Inhalte auf immersive, interaktive und intelligente Weise zu vermitteln, können XR-Technologien die Produktivität in Unternehmen steigern und Wachstumschancen eröffnen. Obwohl XR-Anwendungen in der Industrie seit mehr als 25 Jahren wissenschaftlich erforscht werden, gelten nach wie vor als unausgereift. Die Hauptgründe dafür sind die zugrundeliegende Komplexität, die Fokussierung der Forschung auf die Untersuchung spezifische Anwendungsszenarien, die unzu-reichende Wirtschaftlichkeit von Einsatzszenarien und das Fehlen von geeigneten Implementierungsmodellen für XR-Technologien. Grundsätzlich wird der Mehrwert von Technologien durch deren Integration in die Wertschöpfungsarchitektur von Geschäftsmodellen freigesetzt. Daher wird in dieser Arbeit eine Methodik für den Einsatz von XR-Technologien in der Wertschöpfung vorgestellt. Das Hauptziel der Methodik ist es, die Identifikation geeigneter Einsatzszenarien zu ermöglichen und mit einem strukturierten Ablauf die Komplexität der Umsetzung zu beherrschen. Um eine ganzheitliche Anwendbarkeit zu ermöglichen, basiert die Methodik auf einem branchen- und ge-schäftsprozessunabhängigen Wertschöpfungsreferenzmodell. Dar-über hinaus bezieht sie sich auf eine ganzheitliche Morphologie von XR-Technologien und folgt einer iterativen Einführungssequenz. Das Wertschöpfungsmodell wird durch ein vorliegendes Potential, eine Wertschöpfungskette, ein Wertschöpfungsnetzwerk, physische und digitale Ressourcen sowie einen durch den Einsatz von XR-Technologien realisierten Mehrwert repräsentiert. XR-Technologien werden durch eine morphologische Struktur mit Anwendungsmerk-malen und erforderlichen technologischen Ressourcen repräsentiert. Die Umsetzung erfolgt in einer iterativen Sequenz, die für den zu-grundeliegenden Kontext anwendbare Methoden der agilen Soft-wareentwicklung beschreibt und relevante Stakeholder berücksich-tigt. Der Schwerpunkt der Methodik liegt auf einem systematischen Ansatz, der universell anwendbar ist und den Endnutzer und das Ökosystem der betrachteten Wertschöpfung berücksichtigt. Um die Methodik zu validieren, wird der Einsatz von XR-Technologien in zwei industriellen Anwendungsfällen unter realen wirtschaftlichen Bedingungen durchgeführt. Die Anwendungsfälle stammen aus unterschiedlichen Branchen, mit unterschiedlichen XR-Technologiemerkmalen sowie unterschiedlichen Formen von Wert-schöpfungsketten, um die universelle Anwendbarkeit der Methodik zu demonstrieren und relevante Herausforderungen bei der Durch-führung eines XR-Technologieeinsatzes aufzuzeigen. Mit Hilfe der vorgestellten Methodik können Unternehmen XR-Technologien zielgerichtet in ihrer Wertschöpfung einsetzen. Sie ermöglicht eine detaillierte Planung der Umsetzung, eine fundierte Auswahl von Anwendungsszenarien, die Bewertung möglicher Her-ausforderungen und Hindernisse sowie die gezielte Einbindung der relevanten Stakeholder. Im Ergebnis wird die Wertschöpfung mit wirtschaftlichem Mehrwert durch XR-Technologien optimiert

Developing digital fieldwork technologies at the British Geological Survey

Geological Surveys are faced with budget constraints and calls for efficiency gains; the effective application of digital techniques is often seen as a route to meeting these demands while increasing the value of outcrop studies and reducing the inherent subsurface uncertainty. The British Geological Survey may be the oldest national Survey in the world (established in 1835); however, developing and implementing new, innovative and efficient technologies for fieldwork is a high priority. Efficient tools for capturing, integrating, manipulating and disseminating outcrop data and information are imperative to enable geoscientists to increase their understanding of geological processes and therefore to reduce subsurface uncertainty and risk. Systems for capturing structured digital field data and for visualizing and interacting with large datasets are increasingly being utilized by geoscientists in the UK and internationally. Augmented reality and unmanned aerial vehicles are amongst the developing technologies being explored for future operational implementation. This paper describes the digital field mapping (BGS·SIGMAmobile) and visualization (GeoVisionary) systems and refers to a case study outlining their contribution to reducing uncertainty and risk in hydrocarbon exploration