Requirement analysis and sensor specifications – First version

In this first version of the deliverable, we make the following contributions: to design the WEKIT capturing platform and the associated experience capturing API, we use a methodology for system engineering that is relevant for different domains such as: aviation, space, and medical and different professions such as: technicians, astronauts, and medical staff. Furthermore, in the methodology, we explore the system engineering process and how it can be used in the project to support the different work packages and more importantly the different deliverables that will follow the current. Next, we provide a mapping of high level functions or tasks (associated with experience transfer from expert to trainee) to low level functions such as: gaze, voice, video, body posture, hand gestures, bio-signals, fatigue levels, and location of the user in the environment. In addition, we link the low level functions to their associated sensors. Moreover, we provide a brief overview of the state-of-the-art sensors in terms of their technical specifications, possible limitations, standards, and platforms. We outline a set of recommendations pertaining to the sensors that are most relevant for the WEKIT project taking into consideration the environmental, technical and human factors described in other deliverables. We recommend Microsoft Hololens (for Augmented reality glasses), MyndBand and Neurosky chipset (for EEG), Microsoft Kinect and Lumo Lift (for body posture tracking), and Leapmotion, Intel RealSense and Myo armband (for hand gesture tracking). For eye tracking, an existing eye-tracking system can be customised to complement the augmented reality glasses, and built-in microphone of the augmented reality glasses can capture the expert’s voice. We propose a modular approach for the design of the WEKIT experience capturing system, and recommend that the capturing system should have sufficient storage or transmission capabilities. Finally, we highlight common issues associated with the use of different sensors. We consider that the set of recommendations can be useful for the design and integration of the WEKIT capturing platform and the WEKIT experience capturing API to expedite the time required to select the combination of sensors which will be used in the first prototype.WEKI

Harnessing the Potential of Optical Communications for the Metaverse

The Metaverse is a digital world that offers an immersive virtual experience. However, the Metaverse applications are bandwidth-hungry and delay-sensitive that require ultrahigh data rates, ultra-low latency, and hyper-intensive computation. To cater for these requirements, optical communication arises as a key pillar in bringing this paradigm into reality. We highlight in this paper the potential of optical communications in the Metaverse. First, we set forth Metaverse requirements in terms of capacity and latency; then, we introduce ultra-high data rates requirements for various Metaverse experiences. Then, we put forward the potential of optical communications to achieve these data rate requirements in backbone, backhaul, fronthaul, and access segments. Both optical fiber and optical wireless communication (OWC) technologies, as well as their current and future expected data rates, are detailed. In addition, we propose a comprehensive set of configurations, connectivity, and equipment necessary for an immersive Metaverse experience. Finally, we identify a set of key enablers and research directions such as analog neuromorphic optical computing, optical intelligent reflective surfaces (IRS), hollow core fiber (HCF), and terahertz (THz)

CoVR: A Large-Scale Force-Feedback Robotic Interface for Non-Deterministic Scenarios in VR

We present CoVR, a novel robotic interface providing strong kinesthetic feedback (100 N) in a room-scale VR arena. It consists of a physical column mounted on a 2D Cartesian ceiling robot (XY displacements) with the capacity of (1) resisting to body-scaled users' actions such as pushing or leaning; (2) acting on the users by pulling or transporting them as well as (3) carrying multiple potentially heavy objects (up to 80kg) that users can freely manipulate or make interact with each other. We describe its implementation and define a trajectory generation algorithm based on a novel user intention model to support non-deterministic scenarios, where the users are free to interact with any virtual object of interest with no regards to the scenarios' progress. A technical evaluation and a user study demonstrate the feasibility and usability of CoVR, as well as the relevance of whole-body interactions involving strong forces, such as being pulled through or transported.Comment: 10 pages (without references), 14 pages tota

A tangible programming environment model informed by principles of perception and meaning

It is a fundamental Human-Computer Interaction problem to design a tangible programming environment for use by multiple persons that can also be individualised. This problem has its origin in the phenomenon that the meaning an object holds can vary across individuals. The Semiotics Research Domain studies the meaning objects hold. This research investigated a solution based on the user designing aspects of the environment at a time after it has been made operational and when the development team is no longer available to implement the user’s design requirements. Also considered is how objects can be positioned so that the collection of objects is interpreted as a program. I therefore explored how some of the principles of relative positioning of objects, as researched in the domains of Psychology and Art, could be applied to tangible programming environments. This study applied the Gestalt principle of perceptual grouping by proximity to the design of tangible programming environments to determine if a tangible programming environment is possible in which the relative positions of personally meaningful objects define the program. I did this by applying the Design Science Research methodology with five iterations and evaluations involving children. The outcome is a model of a Tangible Programming Environment that includes Gestalt principles and Semiotic theory; Semiotic theory explains that the user can choose a physical representation of the program element that carries personal meaning whereas the Gestalt principle of grouping by proximity predicts that objects can be arranged to appear as if linked to each other.School of ComputingPh. D. (Computer Science

Furthering Service 4.0: Harnessing Intelligent Immersive Environments and Systems

With the increasing complexity of service operations in different industries and more advanced uses of specialized equipment and procedures, the great current challenge for companies is to increase employees' expertise and their ability to maintain and improve service quality. In this regard, Service 4.0 aims to support and promote innovation in service operations using emergent technology. Current technological innovations present a significant opportunity to provide on-site, real-time support for field service professionals in many areas

Variability of grid-cell activity

Action potentials of grid cells in the entorhinal cortex of navigating rodents occur every two seconds on average. If one considers the precise temporal sequence of these events, however, it can be seen that they rarely occur in isolation. In fact, the intervals between successive action potentials can be on the order of a few milliseconds. Mapped to the trajectory of the animal, a clear clustering of the action potentials in space can be observed as well. The places where the density of such events is particularly high are called firing fields and are arranged in a hexagonal grid. Regardless of the cell characteristics, the number of spikes observed on different crossings of a field varies strongly. The time between subsequent field crossings is on the order of seconds. We found out that one cause of spike-count variability is that the exact position of the firing fields is not stable over time. In addition, the shifts of the fields were correlated across simultaneously recorded cells. This kind of non-stationarity in the grid-cell network allows conclusions to be drawn about the functioning of this system. Furthermore, dynamic field locations imply that common methods for data analysis of grid-cell recordings can be problematic. Furthermore, we found out that a subset of grid cells, which have particularly high firing rates when crossing a field, can be associated with a peculiarity in the shape of their action potentials: The spikes of some cells are followed by a short afterdepolarization (DAP). At the same time, we discovered cells with even smaller and extremely stereotypical intervals between their spikes. This group of neurons, however, exhibited less pronounced DAPs. Cells with and without DAP did not differ in their spatial firing behavior. Our results imply that different burst behaviors are not directly related to different types of spatial coding. In addition, we suggest that bursting of grid cells could be altered via the mechanisms of DAP formation. In summary, this work shows how details of neuronal activity on two different time scales provide fundamental insights into the processes of spatial navigation. Untethered firing fields and intermittent silences: Why grid‐cell discharge is so variable - Grid cells in medial entorhinal cortex are notoriously variable in their responses, despite the striking hexagonal arrangement of their spatial firing fields. Indeed, when the animal moves through a firing field, grid cells often fire much more vigorously than predicted or do not fire at all. The source of this trial‐to‐trial variability is not completely understood. By analyzing grid‐cell spike trains from mice running in open arenas and on linear tracks, we characterize the phenomenon of “missed” firing fields using the statistical theory of zero inflation. We find that one major cause of grid‐cell variability lies in the spatial representation itself: firing fields are not as strongly anchored to spatial location as the averaged grid suggests. In addition, grid fields from different cells drift together from trial to trial, regardless of whether the environment is real or virtual, or whether the animal moves in light or darkness. Spatial realignment across trials sharpens the grid representation, yielding firing fields that are more pronounced and significantly narrower. These findings indicate that ensembles of grid cells encode relative position more reliably than absolute position. Spike Afterpotentials Shape the In Vivo Burst Activity of Principal Cells in Medial Entorhinal Cortex - Principal neurons in rodent medial entorhinal cortex (MEC) generate high-frequency bursts during natural behavior. While in vitro studies point to potential mechanisms that could support such burst sequences, it remains unclear whether these mechanisms are effective under in vivo conditions. In this study, we focused on the membrane-potential dynamics immediately following action potentials (APs), as measured in whole-cell recordings from male mice running in virtual corridors (Domnisoru et al., 2013). These afterpotentials consisted either of a hyperpolarization, an extended ramp-like shoulder, or a depolarization reminiscent of depolarizing afterpotentials (DAPs) recorded in vitro in MEC principal neurons. Next, we correlated the afterpotentials with the cells' propensity to fire bursts. All DAP cells with known location resided in Layer II, generated bursts, and their interspike intervals (ISIs) were typically between 5 and 15 ms. The ISI distributions of Layer-II cells without DAPs peaked sharply at around 4 ms and varied only minimally across that group. This dichotomy in burst behavior is explained by cell-group-specific DAP dynamics. The same two groups of bursting neurons also emerged when we clustered extracellular spike-train autocorrelations measured in real 2D arenas (Latuske et al., 2015). Apart from slight variations in grid spacing, no difference in the spatial coding properties of the grid cells across all three groups was discernible. Layer III neurons were only sparsely bursting (SB) and had no DAPs. As various mechanisms for modulating ion-channels underlying DAPs exist, our results suggest that temporal features of MEC activity can be altered while maintaining the cells' overall spatial tuning characteristics