A Tagless Indoor Localization System Based on Capacitive Sensing Technology

Accurate indoor person localization is essential for several services, such as assisted living. We introduce a tagless indoor person localization system based on capacitive sensing and localization algorithms that can determine the location with less than 0.2 m average error in a 3 m × 3 m room and has recall and precision better than 70%. We also discuss the effects of various noise types on the measurements and ways to reduce them using filters suitable for on-sensor implementation to lower communication energy consumption. We also compare the performance of several standard localization algorithms in terms of localization error, recall, precision, and accuracy of detection of the movement trajectory

Passive Electric Field Sensing for Ubiquitous and Environmental Perception

Electric Field Sensing plays an important role in the research branches of Environmental Perception as well as in Ubiquitous Computing. Environmental Perception aims to collect data of the surroundings, while Ubiquitous Computing has the objective of making computing available at any time. This includes the integration of sensors to perceive environmental influences in an unobtrusive way. Electric Field Sensing, also referenced as Capacitive Sensing, is an often used sensing modality in these research fields, for example, to detect the presence of persons or to locate touches and interactions on user interfaces. Electric Field Sensing has a number of advantages over other technologies, such as the fact that Capacitive Sensing does not require direct line-of-sight contact with the object being sensed and that the sensing system can be compact in design. These advantages facilitate high integrability and allow the collection of data as required in Environmental Perception, as well as the invisible incorporation into a user's environment, needed in Ubiquitous Computing. However, disadvantages are often attributed to Capacitive Sensing principles, such as a low sensing range of only a few centimeters and the generation of electric fields, which wastes energy and has several more problems concerning the implementation. As shown in this thesis, this only affects a subset of this sensing technology, namely the subcategory of active capacitive measurements. Therefore, this thesis focuses on the mainly open area of Passive Electric Field Sensing in the context of Ubiquitous Computing and Environmental Perception, as active Capacitive Sensing is an open research field which already gains a lot of attention. The thesis is divided into three main research questions. First, I address the question of whether and how Passive Electric Field Sensing can be made available in a cost-effective and simple manner. To this end, I present various techniques for reducing installation costs and simplifying the handling of these sensor systems. After the question of low-cost applicability, I examine for which applications passive electric field sensor technology is suitable at all. Therefore I present several fields of application where Passive Electric Field Sensing data can be collected. Taking into account the possible fields of application, this work is finally dedicated to the optimization of Passive Electric Field Sensing in these cases of application. For this purpose, different, already known signal processing methods are investigated for their application for Passive Electric Field sensor data. Furthermore, besides these software optimizations, hardware optimizations for the improved use of the technology are presented

Capacitive Sensing and Communication for Ubiquitous Interaction and Environmental Perception

During the last decade, the functionalities of electronic devices within a living environment constantly increased. Besides the personal computer, now tablet PCs, smart household appliances, and smartwatches enriched the technology landscape. The trend towards an ever-growing number of computing systems has resulted in many highly heterogeneous human-machine interfaces. Users are forced to adapt to technology instead of having the technology adapt to them. Gathering context information about the user is a key factor for improving the interaction experience. Emerging wearable devices show the benefits of sophisticated sensors which make interaction more efficient, natural, and enjoyable. However, many technologies still lack of these desirable properties, motivating me to work towards new ways of sensing a user's actions and thus enriching the context. In my dissertation I follow a human-centric approach which ranges from sensing hand movements to recognizing whole-body interactions with objects. This goal can be approached with a vast variety of novel and existing sensing approaches. I focused on perceiving the environment with quasi-electrostatic fields by making use of capacitive coupling between devices and objects. Following this approach, it is possible to implement interfaces that are able to recognize gestures, body movements and manipulations of the environment at typical distances up to 50cm. These sensors usually have a limited resolution and can be sensitive to other conductive objects or electrical devices that affect electric fields. The technique allows for designing very energy-efficient and high-speed sensors that can be deployed unobtrusively underneath any kind of non-conductive surface. Compared to other sensing techniques, exploiting capacitive coupling also has a low impact on a user's perceived privacy. In this work, I also aim at enhancing the interaction experience with new perceptional capabilities based on capacitive coupling. I follow a bottom-up methodology and begin by presenting two low-level approaches for environmental perception. In order to perceive a user in detail, I present a rapid prototyping toolkit for capacitive proximity sensing. The prototyping toolkit shows significant advancements in terms of temporal and spatial resolution. Due to some limitations, namely the inability to determine the identity and fine-grained manipulations of objects, I contribute a generic method for communications based on capacitive coupling. The method allows for designing highly interactive systems that can exchange information through air and the human body. I furthermore show how human body parts can be recognized from capacitive proximity sensors. The method is able to extract multiple object parameters and track body parts in real-time. I conclude my thesis with contributions in the domain of context-aware devices and explicit gesture-recognition systems

ExerTrack - Towards Smart Surfaces to Track Exercises

The concept of the quantified self has gained popularity in recent years with the hype of miniaturized gadgets to monitor vital fitness levels. Smartwatches or smartphone apps and other fitness trackers are overwhelming the market. Most aerobic exercises such as walking, running, or cycling can be accurately recognized using wearable devices. However whole-body exercises such as push-ups, bridges, and sit-ups are performed on the ground and thus cannot be precisely recognized by wearing only one accelerometer. Thus, a floor-based approach is preferred for recognizing whole-body activities. Computer vision techniques on image data also report high recognition accuracy; however, the presence of a camera tends to raise privacy issues in public areas. Therefore, we focus on combining the advantages of ubiquitous proximity-sensing with non-optical sensors to preserve privacy in public areas and maintain low computation cost with a sparse sensor implementation. Our solution is the ExerTrack, an off-the-shelf sports mat equipped with eight sparsely distributed capacitive proximity sensors to recognize eight whole-body fitness exercises with a user-independent recognition accuracy of 93.5 % and a user-dependent recognition accuracy of 95.1 % based on a test study with 9 participants each performing 2 full sessions. We adopt a template-based approach to count repetitions and reach a user-independent counting accuracy of 93.6 %. The final model can run on a Raspberry Pi 3 in real time. This work includes data-processing of our proposed system and model selection to improve the recognition accuracy and data augmentation technique to regularize the network

Neural Networks for Indoor Human Activity Reconstructions

Low cost, ubiquitous, tagless, and privacy aware indoor monitoring is essential to many existing or future applications, such as assisted living of elderly persons. We explore how well different types of neural networks in basic configurations can extract location and movement information from noisy experimental data (with both high-pitch and slow drift noise) obtained from capacitive sensors operating in loading mode at ranges much longer that the diagonal of their plates. Through design space exploration, we optimize and analyze the location and trajectory tracking inference performance of multilayer perceptron (MLP), autoregressive feedforward, 1D Convolutional (1D-CNN), and Long-Short Term Memory (LSTM) neural networks on experimental data collected using four capacitive sensors with 16 cm x 16 cm plates deployed on the boundaries of a 3 m x 3 m open space in our laboratory. We obtain the minimum error using a 1D-CNN [0.251 m distance Root Mean Square Error (RMSE) and 0.307 m Average Distance Error (ADE)] and the smoothest trajectory inference using an LSTM, albeit with higher localization errors (0.281 m RMSE and 0.326 m ADE). 1D Convolutional and window-based neural networks have best inference accuracy and smoother trajectory reconstruction. LSTMs seem to infer best the person movement dynamics

Child's play: activity recognition for monitoring children's developmental progress with augmented toys

The way in which infants play with objects can be indicative of their developmental progress and may serve as an early indicator for developmental delays. However, the observation of children interacting with toys for the purpose of quantitative analysis can be a difficult task. To better quantify how play may serve as an early indicator, researchers have conducted retrospective studies examining the differences in object play behaviors among infants. However, such studies require that researchers repeatedly inspect videos of play often at speeds much slower than real-time to indicate points of interest. The research presented in this dissertation examines whether a combination of sensors embedded within toys and automatic pattern recognition of object play behaviors can help expedite this process. For my dissertation, I developed the Child'sPlay system which uses augmented toys and statistical models to automatically provide quantitative measures of object play interactions, as well as, provide the PlayView interface to view annotated play data for later analysis. In this dissertation, I examine the hypothesis that sensors embedded in objects can provide sufficient data for automatic recognition of certain exploratory, relational, and functional object play behaviors in semi-naturalistic environments and that a continuum of recognition accuracy exists which allows automatic indexing to be useful for retrospective review. I designed several augmented toys and used them to collect object play data from more than fifty play sessions. I conducted pattern recognition experiments over this data to produce statistical models that automatically classify children's object play behaviors. In addition, I conducted a user study with twenty participants to determine if annotations automatically generated from these models help improve performance in retrospective review tasks. My results indicate that these statistical models increase user performance and decrease perceived effort when combined with the PlayView interface during retrospective review. The presence of high quality annotations are preferred by users and promotes an increase in the effective retrieval rates of object play behaviors.Ph.D.Committee Chair: Starner, Thad E.; Committee Co-Chair: Abowd, Gregory D.; Committee Member: Arriaga, Rosa; Committee Member: Jackson, Melody Moore; Committee Member: Lukowicz, Paul; Committee Member: Rehg, James M

Hardware for recognition of human activities: a review of smart home and AAL related technologies

Activity recognition (AR) from an applied perspective of ambient assisted living (AAL) and smart homes (SH) has become a subject of great interest. Promising a better quality of life, AR applied in contexts such as health, security, and energy consumption can lead to solutions capable of reaching even the people most in need. This study was strongly motivated because levels of development, deployment, and technology of AR solutions transferred to society and industry are based on software development, but also depend on the hardware devices used. The current paper identifies contributions to hardware uses for activity recognition through a scientific literature review in the Web of Science (WoS) database. This work found four dominant groups of technologies used for AR in SH and AAL—smartphones, wearables, video, and electronic components—and two emerging technologies: Wi-Fi and assistive robots. Many of these technologies overlap across many research works. Through bibliometric networks analysis, the present review identified some gaps and new potential combinations of technologies for advances in this emerging worldwide field and their uses. The review also relates the use of these six technologies in health conditions, health care, emotion recognition, occupancy, mobility, posture recognition, localization, fall detection, and generic activity recognition applications. The above can serve as a road map that allows readers to execute approachable projects and deploy applications in different socioeconomic contexts, and the possibility to establish networks with the community involved in this topic. This analysis shows that the research field in activity recognition accepts that specific goals cannot be achieved using one single hardware technology, but can be using joint solutions, this paper shows how such technology works in this regard

Application and validation of capacitive proximity sensing systems in smart environments

Smart environments feature a number of computing and sensing devices that support occupants in performing their tasks. In the last decades there has been a multitude of advances in miniaturizing sensors and computers, while greatly increasing their performance. As a result new devices are introduced into our daily lives that have a plethora of functions. Gathering information about the occupants is fundamental in adapting the smart environment according to preference and situation. There is a large number of different sensing devices available that can provide information about the user. They include cameras, accelerometers, GPS, acoustic systems, or capacitive sensors. The latter use the properties of an electric field to sense presence and properties of conductive objects within range. They are commonly employed in finger-controlled touch screens that are present in billions of devices. A less common variety is the capacitive proximity sensor. It can detect the presence of the human body over a distance, providing interesting applications in smart environments. Choosing the right sensor technology is an important decision in designing a smart environment application. Apart from looking at previous use cases, this process can be supported by providing more formal methods. In this work I present a benchmarking model that is designed to support this decision process for applications in smart environments. Previous benchmarks for pervasive systems have been adapted towards sensors systems and include metrics that are specific for smart environments. Based on distinct sensor characteristics, different ratings are used as weighting factors in calculating a benchmarking score. The method is verified using popularity matching in two scientific databases. Additionally, there are extensions to cope with central tendency bias and normalization with regards to average feature rating. Four relevant application areas are identified by applying this benchmark to applications in smart environments and capacitive proximity sensors. They are indoor localization, smart appliances, physiological sensing and gesture interaction. Any application area has a set of challenges regarding the required sensor technology, layout of the systems, and processing that can be tackled using various new or improved methods. I will present a collection of existing and novel methods that support processing data generated by capacitive proximity sensors. These are in the areas of sparsely distributed sensors, model-driven fitting methods, heterogeneous sensor systems, image-based processing and physiological signal processing. To evaluate the feasibility of these methods, several prototypes have been created and tested for performance and usability. Six of them are presented in detail. Based on these evaluations and the knowledge generated in the design process, I am able to classify capacitive proximity sensing in smart environments. This classification consists of a comparison to other popular sensing technologies in smart environments, the major benefits of capacitive proximity sensors, and their limitations. In order to support parties interested in developing smart environment applications using capacitive proximity sensors, I present a set of guidelines that support the decision process from technology selection to choice of processing methods

A Computer-Based Touch-Less 3D Controller Using Capacitive Sensing Method

This paper focuses on the application of touch-less interaction between human and computer using capacitive sensing technique. A computer–based analysis for touch-less 3D controller using capacitive sensing method in [1] is developed. In this project, Arduino UNO is used as a microcontroller to bridge the interface connection between the sensor hardware and the computer. This method uses capacitive based sensor as the main component to sense the gesture movement near it. The capacitive based sensing depends on the duration to charge a capacitor (known as the time constant). By placing an object within the electric field of a capacitor, it will immediately affect the capacitance value and it will correspond to the time constant. In the final analysis, the touch-less hardware will be linked to MATLAB software to study its characteristic and behavior. Using the data obtained from the analysis, a touch-less control from the hardware will control the computer keyboard. To show its additional functionality, a Google Earth program will display the ability of the touch-less interface

A Computer-based Touch-less 3D Controller Using Capacitive Sensing Method

This paper focuses on the application of touch-less interaction between human and computer using capacitive sensing technique. A computer–based analysis for touch-less 3D controller using capacitive sensing method in [1] is developed. In this project, Arduino UNO is used as a microcontroller to bridge the interface connection between the sensor hardware and the computer. This method uses capacitive based sensor as the main component to sense the gesture movement near it. The capacitive based sensing depends on the duration to charge a capacitor (known as the time constant). By placing an object within the electric field of a capacitor, it will immediately affect the capacitance value and it will correspond to the time constant. In the final analysis, the touch-less hardware will be linked to MATLAB software to study its characteristic and behavior. Using the data obtained from the analysis, a touch-less control from the hardware will control the computer keyboard. To show its additional functionality, a Google Earth program will display the ability of the touch-less interface