Learning Longterm Representations for Person Re-Identification Using Radio Signals

Person Re-Identification (ReID) aims to recognize a person-of-interest across different places and times. Existing ReID methods rely on images or videos collected using RGB cameras. They extract appearance features like clothes, shoes, hair, etc. Such features, however, can change drastically from one day to the next, leading to inability to identify people over extended time periods. In this paper, we introduce RF-ReID, a novel approach that harnesses radio frequency (RF) signals for longterm person ReID. RF signals traverse clothes and reflect off the human body; thus they can be used to extract more persistent human-identifying features like body size and shape. We evaluate the performance of RF-ReID on longitudinal datasets that span days and weeks, where the person may wear different clothes across days. Our experiments demonstrate that RF-ReID outperforms state-of-the-art RGB-based ReID approaches for long term person ReID. Our results also reveal two interesting features: First since RF signals work in the presence of occlusions and poor lighting, RF-ReID allows for person ReID in such scenarios. Second, unlike photos and videos which reveal personal and private information, RF signals are more privacy-preserving, and hence can help extend person ReID to privacy-concerned domains, like healthcare.Comment: CVPR 2020. The first three authors contributed equally to this pape

In-Home Daily-Life Captioning Using Radio Signals

This paper aims to caption daily life --i.e., to create a textual description of people's activities and interactions with objects in their homes. Addressing this problem requires novel methods beyond traditional video captioning, as most people would have privacy concerns about deploying cameras throughout their homes. We introduce RF-Diary, a new model for captioning daily life by analyzing the privacy-preserving radio signal in the home with the home's floormap. RF-Diary can further observe and caption people's life through walls and occlusions and in dark settings. In designing RF-Diary, we exploit the ability of radio signals to capture people's 3D dynamics, and use the floormap to help the model learn people's interactions with objects. We also use a multi-modal feature alignment training scheme that leverages existing video-based captioning datasets to improve the performance of our radio-based captioning model. Extensive experimental results demonstrate that RF-Diary generates accurate captions under visible conditions. It also sustains its good performance in dark or occluded settings, where video-based captioning approaches fail to generate meaningful captions. For more information, please visit our project webpage: http://rf-diary.csail.mit.eduComment: ECCV 2020. The first two authors contributed equally to this pape

A Benchmark of Video-Based Clothes-Changing Person Re-Identification

Person re-identification (Re-ID) is a classical computer vision task and has achieved great progress so far. Recently, long-term Re-ID with clothes-changing has attracted increasing attention. However, existing methods mainly focus on image-based setting, where richer temporal information is overlooked. In this paper, we focus on the relatively new yet practical problem of clothes-changing video-based person re-identification (CCVReID), which is less studied. We systematically study this problem by simultaneously considering the challenge of the clothes inconsistency issue and the temporal information contained in the video sequence for the person Re-ID problem. Based on this, we develop a two-branch confidence-aware re-ranking framework for handling the CCVReID problem. The proposed framework integrates two branches that consider both the classical appearance features and cloth-free gait features through a confidence-guided re-ranking strategy. This method provides the baseline method for further studies. Also, we build two new benchmark datasets for CCVReID problem, including a large-scale synthetic video dataset and a real-world one, both containing human sequences with various clothing changes. We will release the benchmark and code in this work to the public

SoK: Inference Attacks and Defenses in Human-Centered Wireless Sensing

Human-centered wireless sensing aims to understand the fine-grained environment and activities of a human using the diverse wireless signals around her. The wireless sensing community has demonstrated the superiority of such techniques in many applications such as smart homes, human-computer interactions, and smart cities. Like many other technologies, wireless sensing is also a double-edged sword. While the sensed information about a human can be used for many good purposes such as enhancing life quality, an adversary can also abuse it to steal private information about the human (e.g., location, living habits, and behavioral biometric characteristics). However, the literature lacks a systematic understanding of the privacy vulnerabilities of wireless sensing and the defenses against them. In this work, we aim to bridge this gap. First, we propose a framework to systematize wireless sensing-based inference attacks. Our framework consists of three key steps: deploying a sniffing device, sniffing wireless signals, and inferring private information. Our framework can be used to guide the design of new inference attacks since different attacks can instantiate these three steps differently. Second, we propose a defense-in-depth framework to systematize defenses against such inference attacks. The prevention component of our framework aims to prevent inference attacks via obfuscating the wireless signals around a human, while the detection component aims to detect and respond to attacks. Third, based on our attack and defense frameworks, we identify gaps in the existing literature and discuss future research directions

Towards Practical and Secure Channel Impulse Response-based Physical Layer Key Generation

Der derzeitige Trend hin zu “smarten” Geräten bringt eine Vielzahl an Internet-fähigen und verbundenen Geräten mit sich. Die entsprechende Kommunikation dieser Geräte muss zwangsläufig durch geeignete Maßnahmen abgesichert werden, um die datenschutz- und sicherheitsrelevanten Anforderungen an die übertragenen Informationen zu erfüllen. Jedoch zeigt die Vielzahl an sicherheitskritischen Vorfällen im Kontext von “smarten” Geräten und des Internets der Dinge auf, dass diese Absicherung der Kommunikation derzeit nur unzureichend umgesetzt wird. Die Ursachen hierfür sind vielfältig: so werden essentielle Sicherheitsmaßnahmen im Designprozess mitunter nicht berücksichtigt oder auf Grund von Preisdruck nicht realisiert. Darüber hinaus erschwert die Beschaffenheit der eingesetzten Geräte die Anwendung klassischer Sicherheitsverfahren. So werden in diesem Kontext vorrangig stark auf Anwendungsfälle zugeschnittene Lösungen realisiert, die auf Grund der verwendeten Hardware meist nur eingeschränkte Rechen- und Energieressourcen zur Verfügung haben. An dieser Stelle können die Ansätze und Lösungen der Sicherheit auf physikalischer Schicht (physical layer security, PLS) eine Alternative zu klassischer Kryptografie bieten. Im Kontext der drahtlosen Kommunikation können hier die Eigenschaften des Übertragungskanals zwischen zwei legitimen Kommunikationspartnern genutzt werden, um Sicherheitsprimitive zu implementieren und damit Sicherheitsziele zu realisieren. Konkret können etwa reziproke Kanaleigenschaften verwendet werden, um einen Vertrauensanker in Form eines geteilten, symmetrischen Geheimnisses zu generieren. Dieses Verfahren wird Schlüsselgenerierung basierend auf Kanalreziprozität (channel reciprocity based key generation, CRKG) genannt. Auf Grund der weitreichenden Verfügbarkeit wird dieses Verfahren meist mit Hilfe der Kanaleigenschaft des Empfangsstärkenindikators (received signal strength indicator, RSSI) realisiert. Dies hat jedoch den Nachteil, dass alle physikalischen Kanaleigenschaften auf einen einzigen Wert heruntergebrochen werden und somit ein Großteil der verfügbaren Informationen vernachlässigt wird. Dem gegenüber steht die Verwendung der vollständigen Kanalzustandsinformationen (channel state information, CSI). Aktuelle technische Entwicklungen ermöglichen es zunehmend, diese Informationen auch in Alltagsgeräten zur Verfügung zu stellen und somit für PLS weiterzuverwenden. In dieser Arbeit analysieren wir Fragestellungen, die sich aus einem Wechsel hin zu CSI als verwendetes Schlüsselmaterial ergeben. Konkret untersuchen wir CSI in Form von Ultrabreitband-Kanalimpulsantworten (channel impulse response, CIR). Für die Untersuchungen haben wir initial umfangreiche Messungen vorgenommen und damit analysiert, in wie weit die grundlegenden Annahmen von PLS und CRKG erfüllt sind und die CIRs sich grundsätzlich für die Schlüsselgenerierung eignen. Hier zeigen wir, dass die CIRs der legitimen Kommunikationspartner eine höhere Ähnlichkeit als die eines Angreifers aufzeigen und das somit ein Vorteil gegenüber diesem auf der physikalischen Schicht besteht, der für die Schlüsselgenerierung ausgenutzt werden kann. Basierend auf den Ergebnissen der initialen Untersuchung stellen wir dann grundlegende Verfahren vor, die notwendig sind, um die Ähnlichkeit der legitimen Messungen zu verbessern und somit die Schlüsselgenerierung zu ermöglichen. Konkret werden Verfahren vorgestellt, die den zeitlichen Versatz zwischen reziproken Messungen entfernen und somit die Ähnlichkeit erhöhen, sowie Verfahren, die das in den Messungen zwangsläufig vorhandene Rauschen entfernen. Gleichzeitig untersuchen wir, inwieweit die getroffenen fundamentalen Sicherheitsannahmen aus Sicht eines Angreifers erfüllt sind. Zu diesem Zweck präsentieren, implementieren und analysieren wir verschiedene praktische Angriffsmethoden. Diese Verfahren umfassen etwa Ansätze, bei denen mit Hilfe von deterministischen Kanalmodellen oder durch ray tracing versucht wird, die legitimen CIRs vorherzusagen. Weiterhin untersuchen wir Machine Learning Ansätze, die darauf abzielen, die legitimen CIRs direkt aus den Beobachtungen eines Angreifers zu inferieren. Besonders mit Hilfe des letzten Verfahrens kann hier gezeigt werden, dass große Teile der CIRs deterministisch vorhersagbar sind. Daraus leitet sich der Schluss ab, dass CIRs nicht ohne adäquate Vorverarbeitung als Eingabe für Sicherheitsprimitive verwendet werden sollten. Basierend auf diesen Erkenntnissen entwerfen und implementieren wir abschließend Verfahren, die resistent gegen die vorgestellten Angriffe sind. Die erste Lösung baut auf der Erkenntnis auf, dass die Angriffe aufgrund von vorhersehbaren Teilen innerhalb der CIRs möglich sind. Daher schlagen wir einen klassischen Vorverarbeitungsansatz vor, der diese deterministisch vorhersagbaren Teile entfernt und somit das Eingabematerial absichert. Wir implementieren und analysieren diese Lösung und zeigen ihre Effektivität sowie ihre Resistenz gegen die vorgeschlagenen Angriffe. In einer zweiten Lösung nutzen wir die Fähigkeiten des maschinellen Lernens, indem wir sie ebenfalls in das Systemdesign einbringen. Aufbauend auf ihrer starken Leistung bei der Mustererkennung entwickeln, implementieren und analysieren wir eine Lösung, die lernt, die zufälligen Teile aus den rohen CIRs zu extrahieren, durch die die Kanalreziprozität definiert wird, und alle anderen, deterministischen Teile verwirft. Damit ist nicht nur das Schlüsselmaterial gesichert, sondern gleichzeitig auch der Abgleich des Schlüsselmaterials, da Differenzen zwischen den legitimen Beobachtungen durch die Merkmalsextraktion effizient entfernt werden. Alle vorgestellten Lösungen verzichten komplett auf den Austausch von Informationen zwischen den legitimen Kommunikationspartnern, wodurch der damit verbundene Informationsabfluss sowie Energieverbrauch inhärent vermieden wird

Research on recognition algorithm for gesture page turning based on wireless sensing

When a human body moves within the coverage range of Wi-Fi signals, the reflected Wi-Fi signals by the various parts of the human body change the propagation path, so analysis of the channel state data can achieve the perception of the human motion. By extracting the Channel State Information (CSI) related to human motion from the Wi-Fi signals and analyzing it with the introduced machine learning classification algorithm, the human motion in the spatial environment can be perceived. On the basis of this theory, this paper proposed an algorithm of human behavior recognition based on CSI wireless sensing to realize deviceless and over-the-air slide turning. This algorithm collects the environmental information containing upward or downward wave in a conference room scene, uses the local outlier factor detection algorithm to segment the actions, and then the time domain features are extracted to train Support Vector Machine (SVM) and eXtreme Gradient Boosting (XGBoost) classification modules. The experimental results show that the average accuracy of the XGBoost module sensing slide flipping can reach 94%, and the SVM module can reach 89%, so the module could be extended to the field of smart classroom and significantly improve speech efficiency

Approaches to Non-Intrusive Load Monitoring (NILM) in the Home

When designing and implementing an intelligent energy conservation system for the home, it is essential to have insight into the activities and actions of the occupants. In particular, it is important to understand what appliances are being used and when. In the computational sustainability research community this is known as load disaggregation or Non-Intrusive Load Monitoring (NILM). NILM is a foundational algorithm that can disaggregate a home’s power usage into the individual appliances that are running, identify energy conservation opportunities. This depth report will focus on NILM algorithms, their use and evaluation. We will examine and evaluate the anatomy of NILM, looking at techniques using load monitoring, event detection, feature ex- traction, classification, and accuracy measurement.&nbsp

Employing multi-modal sensors for personalised smart home health monitoring.

Smart home systems are employed worldwide for a variety of automated monitoring tasks. FITsense is a system that performs personalised smart home health monitoring using sensor data. In this thesis, we expand upon this system by identifying the limits of health monitoring using simple IoT sensors, and establishing deployable solutions for new rich sensing technologies. The FITsense system collects data from FitHomes and generates behavioural insights for health monitoring. To allow the system to expand to arbitrary home layouts, sensing applications must be delivered while relying on sparse "ground truth" data. An enhanced data representation was tested for improving activity recognition performance by encoding observed temporal dependencies. Experiments showed an improvement in activity recognition accuracy over baseline data representations with standard classifiers. Channel State Information (CSI) was chosen as our rich sensing technology for its ambient nature and potential deployability. We developed a novel Python toolkit, called CSIKit, to handle various CSI software implementations, including automatic detection for off-the-shelf CSI formats. Previous researchers proposed a method to address AGC effects on COTS CSI hardware, which we tested and found to improve correlation with a baseline without AGC. This implementation was included in the public release of CSIKit. Two sensing applications were delivered using CSIKit to demonstrate its functionality. Our statistical approach to motion detection with CSI data showed a 32% increase in accuracy over an infrared sensor-based solution using data from 2 unique environments. We also demonstrated the first CSI activity recognition application on a Raspberry Pi 4, which achieved an accuracy of 92% with 11 activity classes. An application was then trained to support movement detection using data from all COTS CSI hardware. This was combined with our signal divider implementation to compare CSI wireless and sensing performance characteristics. The IWL5300 exhibited the most consistent wireless performance, while the ESP32 was found to produce viable CSI data for sensing applications. This establishes the ESP32 as a low-cost high-value hardware solution for CSI sensing. To complete this work, an in-home study was performed using real-world sensor data. An ESP32-based CSI sensor was developed to be integrated into our IoT network. This sensor was tested in a FitHome environment to identify how the data from our existing simple sensors could aid sensor development. We performed an experiment to demonstrate that annotations for CSI data could be gathered with infrared motion sensors. Results showed that our new CSI sensor collected real-world data of similar utility to that collected manually in a controlled environment

Walking Speed Detection from 5G prototype System

While most RF-sensing approaches proposed in the literature rely on short-distance indoor point-to-point instrumentation, actual large-scale installation of RF sensing suggests the use of ubiquitously available cellular systems. In particular, the 5th generation of the wireless communication standard (5G) is envisioned as a universal communication means also for Internet of Things devices. This thesis presents an investigation of device-free environmental perception capabilities in a 5G prototype system in two cases; walking speed and human presence detection, and elaborate a comparison with the former case and acceleration sensing analysis. This thesis attempts to analyze the perception capabilities of 5G system in order to recognize human mostly common activities and presence detection near transceiver devices which the instrumentation exploits a device-free system capable of detect activities without carrying devices capitalizing on environmental RF-noise. This is done via the study of existing and related literature. After that, the implementation and evaluation of walking speed and presence detection is described in details. In addition, evaluation consists of utilizing a prototypical 5G system with 52 OFDM carriers over 12.48 MHz bandwidth at 3.45 GHz, which we consider the impact of the number and choice of channels and compare the recognition performance with acceleration-based sensing. It was concluded that in realistic settings with five subjects, accurate recognition of activities and environmental situations can be a reliable implicit service of future 5G installations