Human Digital Twin: A Survey

Digital twin has recently attracted growing attention, leading to intensive research and applications. Along with this, a new research area, dubbed as "human digital twin" (HDT), has emerged. Similar to the conception of digital twin, HDT is referred to as the replica of a physical-world human in the digital world. Nevertheless, HDT is much more complicated and delicate compared to digital twins of any physical systems and processes, due to humans' dynamic and evolutionary nature, including physical, behavioral, social, physiological, psychological, cognitive, and biological dimensions. Studies on HDT are limited, and the research is still in its infancy. In this paper, we first examine the inception, development, and application of the digital twin concept, providing a context within which we formally define and characterize HDT based on the similarities and differences between digital twin and HDT. Then we conduct an extensive literature review on HDT research, analyzing underpinning technologies and establishing typical frameworks in which the core HDT functions or components are organized. Built upon the findings from the above work, we propose a generic architecture for the HDT system and describe the core function blocks and corresponding technologies. Following this, we present the state of the art of HDT technologies and applications in the healthcare, industry, and daily life domain. Finally, we discuss various issues related to the development of HDT and point out the trends and challenges of future HDT research and development

A Review of Verbal and Non-Verbal Human-Robot Interactive Communication

In this paper, an overview of human-robot interactive communication is presented, covering verbal as well as non-verbal aspects of human-robot interaction. Following a historical introduction, and motivation towards fluid human-robot communication, ten desiderata are proposed, which provide an organizational axis both of recent as well as of future research on human-robot communication. Then, the ten desiderata are examined in detail, culminating to a unifying discussion, and a forward-looking conclusion

Immersive Participation:Futuring, Training Simulation and Dance and Virtual Reality

Dance knowledge can inform the development of scenario design in immersive digital simulation environments by strengthening a participant’s capacity to learn through the body. This study engages with processes of participatory practice that question how the transmission and transfer of dance knowledge/embodied knowledge in immersive digital environments is activated and applied in new contexts. These questions are relevant in both arts and industry and have the potential to add value and knowledge through crossdisciplinary collaboration and exchange. This thesis consists of three different research projects all focused on observation, participation, and interviews with experts on embodiment in digital simulation. The projects were chosen to provide a range of perspectives across dance, industry and futures studies. Theories of embodied cognition, in particular the notions of the extended body, distributed cognition, enactment and mindfulness, offer critical lenses through which to explore the relationship of embodied integration and participation within immersive digital environments. These areas of inquiry lead to the consideration of how language from the field of computer science can assist in describing somatic experience in digital worlds through a discussion of the emerging concepts of mindfulness, wayfinding, guided movement and digital kinship. These terms serve as an example of how the mutability of language became part of the process as terms applied in disparate disciplines were understood within varying contexts. The analytic tools focus on applying a posthuman view, speculation through a futures ethnography, and a cognitive ethnographical approach to my research project. These approaches allowed me to examine an ecology of practices in order to identify methods and processes that can facilitate the transmission and transfer of embodied knowledge within a community of practice. The ecological components include dance, healthcare, transport, education and human/computer interaction. These fields drove the data collection from a range of sources including academic papers, texts, specialists’ reports, scientific papers, interviews and conversations with experts and artists.The aim of my research is to contribute both a theoretical and a speculative understanding of processes, as well as tools applicable in the transmission of embodied knowledge in virtual dance and arts environments as well as digital simulation across industry. Processes were understood theoretically through established studies in embodied cognition applied to workbased training, reinterpreted through my own movement study. Futures methodologies paved the way for speculative processes and analysis. Tools to choreograph scenario design in immersive digital environments were identified through the recognition of cross purpose language such as mindfulness, wayfinding, guided movement and digital kinship. Put together, the major contribution of this research is a greater understanding of the value of dance knowledge applied to simulation developed through theoretical and transformational processes and creative tools

SALSA: A Novel Dataset for Multimodal Group Behavior Analysis

Studying free-standing conversational groups (FCGs) in unstructured social settings (e.g., cocktail party ) is gratifying due to the wealth of information available at the group (mining social networks) and individual (recognizing native behavioral and personality traits) levels. However, analyzing social scenes involving FCGs is also highly challenging due to the difficulty in extracting behavioral cues such as target locations, their speaking activity and head/body pose due to crowdedness and presence of extreme occlusions. To this end, we propose SALSA, a novel dataset facilitating multimodal and Synergetic sociAL Scene Analysis, and make two main contributions to research on automated social interaction analysis: (1) SALSA records social interactions among 18 participants in a natural, indoor environment for over 60 minutes, under the poster presentation and cocktail party contexts presenting difficulties in the form of low-resolution images, lighting variations, numerous occlusions, reverberations and interfering sound sources; (2) To alleviate these problems we facilitate multimodal analysis by recording the social interplay using four static surveillance cameras and sociometric badges worn by each participant, comprising the microphone, accelerometer, bluetooth and infrared sensors. In addition to raw data, we also provide annotations concerning individuals' personality as well as their position, head, body orientation and F-formation information over the entire event duration. Through extensive experiments with state-of-the-art approaches, we show (a) the limitations of current methods and (b) how the recorded multiple cues synergetically aid automatic analysis of social interactions. SALSA is available at http://tev.fbk.eu/salsa.Comment: 14 pages, 11 figure

Lichtsuchende: exploring the emergence of a cybernetic society

In this paper, we describe Lichtsuchende, an interactive installation, built using a society of biologically inspired, cybernetic creatures who exchange light as a source of energy and a means of communication. Visitors are invited to engage with the installation using torches to influence and interact with the phototropic robots. As well as describing the finished piece, we explore some of the issues around creating works based on biologically inspired robots. We present an account of the development of the creatures in order to highlight the gulfs between conceptual ideas of how to allow emergent behaviours and the manners in which they are implemented. We also expose the interrelations and tensions between the needs of the creatures as they emerge and the needs of the creators, to understand the duet between the cyber-organisms and their initiators. Finally, we look at the ways in which creators, robots and visitors are enrolled to perform their functions, so that the network of activity can be woven between all parties

Embodied interaction with visualization and spatial navigation in time-sensitive scenarios

Paraphrasing the theory of embodied cognition, all aspects of our cognition are determined primarily by the contextual information and the means of physical interaction with data and information. In hybrid human-machine systems involving complex decision making, continuously maintaining a high level of attention while employing a deep understanding concerning the task performed as well as its context are essential. Utilizing embodied interaction to interact with machines has the potential to promote thinking and learning according to the theory of embodied cognition proposed by Lakoff. Additionally, the hybrid human-machine system utilizing natural and intuitive communication channels (e.g., gestures, speech, and body stances) should afford an array of cognitive benefits outstripping the more static forms of interaction (e.g., computer keyboard). This research proposes such a computational framework based on a Bayesian approach; this framework infers operator\u27s focus of attention based on the physical expressions of the operators. Specifically, this work aims to assess the effect of embodied interaction on attention during the solution of complex, time-sensitive, spatial navigational problems. Toward the goal of assessing the level of operator\u27s attention, we present a method linking the operator\u27s interaction utility, inference, and reasoning. The level of attention was inferred through networks coined Bayesian Attentional Networks (BANs). BANs are structures describing cause-effect relationships between operator\u27s attention, physical actions and decision-making. The proposed framework also generated a representative BAN, called the Consensus (Majority) Model (CMM); the CMM consists of an iteratively derived and agreed graph among candidate BANs obtained by experts and by the automatic learning process. Finally, the best combinations of interaction modalities and feedback were determined by the use of particular utility functions. This methodology was applied to a spatial navigational scenario; wherein, the operators interacted with dynamic images through a series of decision making processes. Real-world experiments were conducted to assess the framework\u27s ability to infer the operator\u27s levels of attention. Users were instructed to complete a series of spatial-navigational tasks using an assigned pairing of an interaction modality out of five categories (vision-based gesture, glove-based gesture, speech, feet, or body balance) and a feedback modality out of two (visual-based or auditory-based). Experimental results have confirmed that physical expressions are a determining factor in the quality of the solutions in a spatial navigational problem. Moreover, it was found that the combination of foot gestures with visual feedback resulted in the best task performance (p\u3c .001). Results have also shown that embodied interaction-based multimodal interface decreased execution errors that occurred in the cyber-physical scenarios (p \u3c .001). Therefore we conclude that appropriate use of interaction and feedback modalities allows the operators maintain their focus of attention, reduce errors, and enhance task performance in solving the decision making problems

Cybonto: Towards Human Cognitive Digital Twins for Cybersecurity

Cyber defense is reactive and slow. On average, the time-to-remedy is hundreds of times larger than the time-to-compromise. In response to the expanding ever-more-complex threat landscape, Digital Twins (DTs) and particularly Human Digital Twins (HDTs) offer the capability of running massive simulations across multiple knowledge domains. Simulated results may offer insights into adversaries' behaviors and tactics, resulting in better proactive cyber-defense strategies. For the first time, this paper solidifies the vision of DTs and HDTs for cybersecurity via the Cybonto conceptual framework proposal. The paper also contributes the Cybonto ontology, formally documenting 108 constructs and thousands of cognitive-related paths based on 20 time-tested psychology theories. Finally, the paper applied 20 network centrality algorithms in analyzing the 108 constructs. The identified top 10 constructs call for extensions of current digital cognitive architectures in preparation for the DT future.Comment: 6 pages, 3 figures, 1 tabl

Statistical Filtering for Multimodal Mobility Modeling in Cyber Physical Systems

A Cyber-Physical System integrates computations and dynamics of physical processes. It is an engineering discipline focused on technology with a strong foundation in mathematical abstractions. It shares many of these abstractions with engineering and computer science, but still requires adaptation to suit the dynamics of the physical world. In such a dynamic system, mobility management is one of the key issues against developing a new service. For example, in the study of a new mobile network, it is necessary to simulate and evaluate a protocol before deployment in the system. Mobility models characterize mobile agent movement patterns. On the other hand, they describe the conditions of the mobile services. The focus of this thesis is on mobility modeling in cyber-physical systems. A macroscopic model that captures the mobility of individuals (people and vehicles) can facilitate an unlimited number of applications. One fundamental and obvious example is traffic profiling. Mobility in most systems is a dynamic process and small non-linearities can lead to substantial errors in the model. Extensive research activities on statistical inference and filtering methods for data modeling in cyber-physical systems exist. In this thesis, several methods are employed for multimodal data fusion, localization and traffic modeling. A novel energy-aware sparse signal processing method is presented to process massive sensory data. At baseline, this research examines the application of statistical filters for mobility modeling and assessing the difficulties faced in fusing massive multi-modal sensory data. A statistical framework is developed to apply proposed methods on available measurements in cyber-physical systems. The proposed methods have employed various statistical filtering schemes (i.e., compressive sensing, particle filtering and kernel-based optimization) and applied them to multimodal data sets, acquired from intelligent transportation systems, wireless local area networks, cellular networks and air quality monitoring systems. Experimental results show the capability of these proposed methods in processing multimodal sensory data. It provides a macroscopic mobility model of mobile agents in an energy efficient way using inconsistent measurements