Immersive Record and Replay for Lively Virtual Environments

Non-player characters (NPCs) play an important role in virtual en-vironments and make them more lively. I am using recording and replaying techniques in virtual reality which enable a single user to interact with previous recordings of themselves. I conducted a user study that suggested that recordings that are performed by a single user are indistinguishable from multi-user recordings. En-couraged by these results, I would like to use machine learning to improve the existing recording and replaying techniques to create more sophisticated NPCs that can react to the user

Record and Transplay: Partial Checkpointing for Replay Debugging

Software bugs that occur in production are often difficult to reproduce in the lab due to subtle differences in the application environment and nondeterminism. Toward addressing this problem, we present Transplay, a system that captures application software bugs as they occur in production and deterministically reproduces them in a completely different environment, potentially running a different operating system, where the application, its binaries and other support data do not exist. Transplay introduces partial checkpointing, a new mechanism that provides two key properties. It efficiently captures the minimal state necessary to reexecute just the last few moments of the application before it encountered a failure. The recorded state, which typically consists of a few megabytes of data, is used to replay the application without requiring the specific application binaries or the original execution environment. Transplay integrates with existing debuggers to provide facilities such as breakpoints and single-stepping to allow the user to examine the contents of variables and other program state at each source line of the application's replayed execution. We have implemented a Transplay prototype that can record unmodified Linux applications and replay them on different versions of Linux as well as Windows. Experiments with server applications such as the Apache web server show that Transplay can be used in production with modest recording overhead

Development and preliminary evaluation of the VPS ReplaySuite: a virtual double-headed microscope for pathology

BACKGROUND: Advances in computing and telecommunications have resulted in the availability of a range of online tools for use in pathology training and quality assurance. The majority focus on either enabling pathologists to examine and diagnose cases, or providing image archives that serve as reference material. Limited emphasis has been placed on analysing the diagnostic process used by pathologists to reach a diagnosis and using this as a resource for improving diagnostic performance. METHODS: The ReplaySuite is an online pathology software tool that presents archived virtual slide examinations to pathologists in an accessible video-like format, similar to observing examinations with a double-headed microscope. Delivered through a customised web browser, it utilises PHP (Hypertext PreProcessor) to interact with a remote database and retrieve data describing virtual slide examinations, performed using the Virtual Pathology Slide (VPS). To demonstrate the technology and conduct a preliminary evaluation of pathologists opinions on its potential application in pathology training and quality assurance, 70 pathologists were invited to use the application to review their own and other pathologists examinations of 10 needle-core breast biopsies and complete an electronic survey. 9 pathologists participated, and all subsequently completed an exit survey. RESULTS: Of those who replayed an examination by another pathologist, 83.3% (5/6) agreed that replays provided an insight into the examining pathologists diagnosis and 33.3% (2/6) reconsidered their own diagnosis for at least one case. Of those who reconsidered their original diagnosis, all re-classified either concordant with group consensus or original glass slide diagnosis. 77.7% (7/9) of all participants, and all 3 participants who replayed more than 10 examinations stated the ReplaySuite to be of some or great benefit in pathology training and quality assurance. CONCLUSION: Participants conclude the ReplaySuite to be of some or of great potential benefit to pathology training and quality assurance and consider the ReplaySuite to be beneficial in evaluating the diagnostic trace of an examination. The ReplaySuite removes temporal and spatial issues that surround the use of double-headed microscopes by allowing examinations to be reviewed at different times and in different locations to the original examination. While the evaluation set was limited and potentially subject to bias, the response of participants was favourable. Further work is planned to determine whether use of the ReplaySuite can result in improved diagnostic ability

Virtuaalitodellisuuteen perustuva robottikäsivarren ohjausjärjestelmä

Virtual reality (VR) is a rising technology for creating previously unseen human machine interfaces. At the same time, general purpose robotic arms are becoming more common in various use cases. The goal of this thesis is to develop a prototype system, which enables controlling a robotic arm remotely utilizing VR technology. To limit the scope, the system is intended for future research in remote medical applications such as diagnosis and surgery. Furthermore, the system will be used especially for researching safety, security and network aspects. The developed prototype system comprises of a Universal Robots UR3 robotic arm and HTC Vive VR system. To study the functionality of the prototype system, two use cases are described and tested. In the first use case, the robotic arm is used to remotely pick up tools and move them on an operation table. In the second use case, the robotic arm is used for remote medical observations and diagnosis with a video feedback link. The conclusion is that a VR controlled robotic system definitely has potential in medical applications. Several possibilities are already studied in this thesis, but the system would need further improvement to reach the level of safe usability required for such critical applications. Moreover, this kind of control system appears to provide unexpected possibilities, such as continuous identification and training artificial intelligence robotic systems, but further research is required. The system is a holistic combination of mechanical engineering, mechatronics and computer science, which is a source of complexity as the tools, processes and software used in different engineering areas do not always fit together well.Virtuaalitodellisuus (VR) on teknologia, jolla voidaan luoda ennennäkemättömiä rajapintoja koneiden ja ihmisten välille. Samanaikaisesti yleiskäyttöiset robottikäsivarret yleistyvät eri käyttötarkoituksissa. Työn tavoite on kehittää prototyyppijärjestelmä, jossa robottikäsivartta etäohjataan virtuaalitodellisuuden avulla. Prototyyppijärjestelmän käyttötarkoitus on uudenlaisten etänä tapahtuvien lääketieteellisten sovellusten tutkimus. Erityisesti tarkoitus on tutkia turvallisuuteen, tietoturvaan ja tietoverkkoihin liittyviä ongelmia ja ratkaisuja. Prototyyppijärjestelmä koostuu Universal Robots UR3 -robottikäsivarresta sekä HTC Vive -VR-järjestelmästä. Prototyyppijärjestelmän toimintaa tutkitaan kahden käyttötapauksen avulla. Ensimmäisessä käyttötapauksessa robotilla poimitaan ja liikutellaan työkaluja leikkauspöydällä etäohjatusti. Toisessa käyttötapauksessa robottia käytetään etädiagnoosiin ja lääketieteellisten mittausten tekemiseen videolinkin avulla. Työn tulos on, että virtuaalitodellisuuden avulla voidaan etäohjata robottikäsivartta intuitiivisesti ja että tällaisella järjestelmällä on mahdollisuuksia lääketieteellisissä sovelluksissa. Työssä tutkitaan useita erilaisia mahdollisuuksia, mutta järjestelmä ei vielä ole riittävän kehittynyt turvallisesti käytettäväksi tällaisissa kriittisissä sovelluksissa. Lisäksi järjestelmä näyttää mahdollistavan odottamattomia lisäominaisuuksia etäohjauksen lisäksi. Esimerkiksi jatkuva käyttäjän tunnistus ja keinoälyn opettaminen ovat mahdollisia jatkotutkimusaiheita. Kehitetty järjestelmä yhdistää konetekniikkaa, mekatroniikkaa ja tietojenkäsittelytiedettä, mikä osaltaan lisää järjestelmän monimutkaisuutta, sillä eri aloilla käytetyt menetelmät, ohjelmistot ja työkalut eivät sovi aina yhteen

Doctor of Philosophy

dissertationA modern software system is a composition of parts that are themselves highly complex: operating systems, middleware, libraries, servers, and so on. In principle, compositionality of interfaces means that we can understand any given module independently of the internal workings of other parts. In practice, however, abstractions are leaky, and with every generation, modern software systems grow in complexity. Traditional ways of understanding failures, explaining anomalous executions, and analyzing performance are reaching their limits in the face of emergent behavior, unrepeatability, cross-component execution, software aging, and adversarial changes to the system at run time. Deterministic systems analysis has a potential to change the way we analyze and debug software systems. Recorded once, the execution of the system becomes an independent artifact, which can be analyzed offline. The availability of the complete system state, the guaranteed behavior of re-execution, and the absence of limitations on the run-time complexity of analysis collectively enable the deep, iterative, and automatic exploration of the dynamic properties of the system. This work creates a foundation for making deterministic replay a ubiquitous system analysis tool. It defines design and engineering principles for building fast and practical replay machines capable of capturing complete execution of the entire operating system with an overhead of several percents, on a realistic workload, and with minimal installation costs. To enable an intuitive interface of constructing replay analysis tools, this work implements a powerful virtual machine introspection layer that enables an analysis algorithm to be programmed against the state of the recorded system through familiar terms of source-level variable and type names. To support performance analysis, the replay engine provides a faithful performance model of the original execution during replay

Record and vPlay: Problem Determination with Virtual Replay Across Heterogeneous Systems

Application down time is one of the major reasons for revenue loss in the modern enterprise. While aggressive release schedules cause frail software to be released, application failures occurring in the field cost millions to the technical support organizations in personnel time. Since developers usually don't have direct access to the field environment for a variety of privacy and security reasons, problems are reproduced, analyzed and fixed in very different lab environments. However, the complexity and diversity of application environments make it difficult to accurately replicate the production environment. The indiscriminate collection of data provided by the bug reports often overwhelm or even mislead the developer. A typical issue requires time consuming rounds of clarifications and interactions with the end user, even after which the issue may not manifest. This dissertation introduces vPlay, a software problem determination system which captures software bugs as they occur in the field into small and self-contained recordings, and allows them to be deterministically reproduced across different operating systems and heterogeneous environments. vPlay makes two key advances over the state of the art. First, the recorded bug can be reproduced in a completely different operating system environment without any kind of dependency on the source. vPlay packages up every piece of data necessary to correctly reproduce the bug on any stateless target machine in the developer environment, without the application, its binaries, and other support data. Second, the data captured by vPlay is small, typically amounting to a few megabytes. vPlay achieves this without requiring changes to the applications, base kernel or hardware. vPlay employs a recording mechanism which provides data level independence between the application and its source environment by adopting a state machine model of the application to capture every piece of state accessed by the application. vPlay minimizes the size of the recording through a new technique called partial checkpointing, to efficiently capture the partial intermediate state of the application required to replay just the last few moments of its execution prior to the failure. The recorded state is saved as a partial checkpoint along with metadata representing the information specific to the source environment, such as call- ing convention used for the system calls on the source system, to make it portable across operating systems. A partial checkpoint is loaded by a partial checkpoint loader, which itself is designed to be portable across different operating systems. Partial checkpointing is combined with a logging mechanism, which monitors the application to identify and record relevant accessed state for root cause analysis and to record application's nondeterministic events. vPlay introduces a new type of virtualization abstraction called vPlay Container, to natively replay an application built for one operating system on another. vPlay Container relies on the self-contained recording produced by vPlay to decouple the application from the target operating system environment in three key areas. The application is decoupled from (1) the address space and its content by transparently fulfilling its memory accesses, (2) the instructions and the processor MMU structures such as segment descriptor tables through a binary translation technique designed specifically for user application code, (3) the operating system interface and its services by abstracting the system call interface through emulation and replay. To facilitate root cause analysis, vPlay Container integrates with a standard debugger to enable the user to set breakpoints and single step the replayed execution of the application to examine the contents of variables and other program state at each source line. We have implemented a vPlay prototype which can record unmodified Linux applications and natively replay them on different versions of Linux as well as Windows. Experiments with several applications including Apache and MySQL show that vPlay can reproduce real bugs and be used in production with modest recording overhead

The VPS ReplaySuite: development and evaluation of a novel, Internet based telepathology tool

The ReplaySuite is a web-based telepathology tool that replicates the doubleheaded microscope environment online, enabling a reviewing pathologist to ‘replay’ an archived virtual slide examination. Examination-tracking data obtained by the Virtual Pathology Slide (VPS) virtual slide viewer is exploited, allowing a remote pathologist to review an examination conducted at a different time and location. This removes temporal and spatial issues associated with double-headed microscopy. In order to conduct a preliminary evaluation of the technology, 9 pathologists used the ReplaySuite to review examination replays and diagnostic data from archived examinations of 10 needlecore breast biopsies. Diagnostically difficult cases were most frequently evaluated, either via diagnostic concordance graphs or examination replays, and all 3 participants who replayed more than 10 examinations stated the ReplaySuite to be of some or great benefit in pathology training and quality assurance. Of those who replayed an examination by another pathologist, 83% (5/6) agreed that replays provided an insight into the examining pathologists diagnosis, and 33% (2/6) reconsidered their own diagnosis for at least one case. Of those who reconsidered their original diagnosis, all reclassified either concordant with group consensus or original glass slide diagnosis. This study demonstrated that the ReplaySuite was of potential benefit in pathology education, however the technology required evaluation in a setting that would facilitate its impact on diagnostic performance. Accordingly, a redeveloped VPS and ReplaySuite were incorporated into the EQUALIS External Quality Assurance (EQA) study in chronic hepatitis staging and grading. During the study, 9 Swedish pathology departments examined and scored digital representations of liver needlecore biopsies during two sessions, with 10 cases per session and two digital slides per case. Between scoring sessions, participants were provided with access to two supplementary electronic resources: the ReplaySuite, and a library of pre-selected reference images. Comparison of concordance with gold standard (KVAST group) scoring before and after electronic resource use facilitated the elucidation of impact on diagnostic performance. Between scoring sessions, participant concordance with KVAST staging increased by 18% (49%-67%), while concordance with KVAST grading increased by 20% (34%-54%). Mean staging un-weighted kappa improved from 0.347 to 0.554 (+0.207), or from ‘fair’ to ‘moderate’ exact agreement with KVAST staging. Linear weighted staging kappa improved from 0.603 to 0.688 (+0.085), indicating close agreement in both sessions. Mean grading unweighted kappa increased from 0.132 to 0.412 (+0.280), or from a ‘poor’ to ‘moderate’ level o f exact agreement with KVAST, while linear weighted kappa improved from 0.328 to 0.624 (+0.295), or from ‘fair’ to ‘good’ level of approximate agreement with KVAST. Subsequent to the EQA scheme, an expert liver pathologist used the ReplaySuite to evaluate study examinations, assessing examination technique and identifying sources of error. Examinations scoring concordant with KVAST were observed to exhibit acceptable examination technique more frequently than discordantly scoring examinations. When grading, 28% (46% - 18%) more concordant than discordant examinations were considered to have viewed sufficient tissue, and at the appropriate magnification. A similar disparity of 24% (59% - 35%) was observed in staging, suggesting that examination technique was important both when determining the degree of necroinflammation within a biopsy, and when ascertaining the extent of fibrosis. In assessing sources of error, the expert pathologist identified a potential source in 50% of grading examinations, with misinterpretation of observed pathology cited in 19%, and missed pathology (oversight) cited in 31% of grading examinations. Of the 41% of staging examinations in which a source was identified, misinterpretation of observed pathology was cited in 20% of examinations, and missed pathology (oversight) in 21% of examinations. This study demonstrated that the use of supplementary electronic resources could result in improvements in diagnostic performance. It also illustrated the significant ‘add on’ value that could be provided by the ReplaySuite in EQA, by providing means to assess not only diagnostic concordance, but also diagnostic technique and identify sources of error. In order to assess Irish trainee pathologist’s perceptions of computer-assisted learning (CAL), a number of commercial systems were utilised to incorporate digital slides into a postgraduate seminar series, and provide subsequent access to seminar digital slides, diagnoses and expert annotations online. All surveyed trainees considered the use of digital slides and expert annotations of benefit in pathology training, and considered the potential implementation of expert examination replays, online self-assessment and the capability to search online for material by organ, diagnosis or pathological feature of benefit. The work described herein illustrates that both expert and trainee pathologists alike consider the use of supplementary electronic resources of benefit in pathology education, and demonstrates that their use can improve diagnostic performance. The ability to evaluate participation in EQA studies via the ReplaySuite provides significant additional value to education schemes, providing a depth of assessment not possible with conventional microscopy

Envirosuite: An Environmentally-Immersive Programming Framework for Wireless Sensor Networks

Networked, embedded sensors allow for an instrumentation of the physical world at unprecedented granularities and from unimagined perspectives. The advent of a ubiquitous sensing era is evident. Yet, sensor network techniques are still far from entering mainstream adoption due to multiple unresolved research challenges, especially due to the high development cost of sensor network applications. Therefore, in this dissertation, we propose to design, implement, and evaluate an environmentally-immersive programming framework, called EnviroSuite, to reduce sensor network software development cost. The goal of our research is to create reusable sensor network development support for the community and reduce the adoption barriers for a broader category of users, ultimately leading to a transition of sensor networks from a research concept to a general-purpose technology available for use for a wide variety of research, government, industry, and everyday purposes. Current sensor network programming practice remains very cumbersome and inefficient for several reasons. First, most existing programming abstractions for sensor networks are either too low-level (thus too tedious and error-prone) or too high-level (unable to support the diversity of sensor network applications). Second, there is no clear separation between application-level programming and system-level programming. A significant concern is the lack of a general middleware library to isolate application developers from low-level details. Finally, testing sensor network systems is particularly challenging. Sensor systems interact heavily with a (non-repeatable) physical environment, making lab experiments not representative and on-site experiments very costly. This dissertation is targeted for a comprehensive solution that addresses all the above-mentioned problems. The EnviroSuite framework consists of (i) a new programming paradigm that exports environment-based abstractions, (ii) critical middleware services that support the abstractions and separate application programmers from tedious, low-level details, and (iii) testing tools geared for in-situ experimenting, debugging, and troubleshooting. First, we introduce a new programming paradigm, called environmentally-immersive programming (EIP), to capture the common characteristics of sensor network applications, the rich, distributed interactions with the physical environment. EIP refers to an object-based programming model in which individual objects represent physical elements in the external environment. It allows the programmer to think directly in terms of physical objects or events of interest. We provide language primitives for programmers to easily implement their environmental tracking and monitoring applications in EIP. A preprocessor translates such EIP code transparently into a library of support middleware services, central to which are object management algorithms, responsible for maintaining a unique mapping between physical and logical objects. The major outcome of sensor networks is observations of the instrumented environment, in other words, sensory data. Implementing an application mainly involves encoding how to generate, store, and collect such data. EIP object abstractions provide simple means for programmers to define how observations of the environment should be made via distributed coordination among multiple nodes, thus simplifying data generation. Yet, the next steps, namely, data storage and collection, remain complicated and fastidious. To isolate programmers from such concerns, we also include in the support library a set of data management services, comprising both network protocols and storage systems to allow data to be collected either in real-time or in a delay-tolerant manner. The final phase in sensor network software development life-cycle is testing, typically performed in-field, where the effects of environmental realities can be studied. However, physical events from the dynamic environment are normally asynchronous and non-repeatable. This lack of repeatability makes the last phase particularly difficult and costly. Hence, it is essential to have the capability to capture and replay sensing events, providing a basis not only for software testing, but also for realistic protocol comparison and parameter tuning. To achieve that, EnviroSuite also provides testing and debugging facilities that enable controllable and repeatable in-field experiments. Finally, to demonstrate the benefits of our framework, we build multiple representative applications upon EnviroSuite, drawn from both tracking systems such as military surveillance, and monitoring systems such as environmental acoustic monitoring. We install these applications into off-the-shelf hardware platforms and physically deploy the hardware into realistic environments. Empirical results collected from such deployments demonstrate the efficacy of EnviroSuite