Multimodal federated learning on IoT data

Federated learning is proposed as an alternative to centralized machine learning since its client-server structure provides better privacy protection and scalability in real-world applications. In many applications, such as smart homes with Internet-of-Things (IoT) devices, local data on clients are generated from different modalities such as sensory, visual, and audio data. Existing federated learning systems only work on local data from a single modality, which limits the scalability of the systems. In this paper, we propose a multimodal and semi-supervised federated learning framework that trains autoencoders to extract shared or correlated representations from different local data modalities on clients. In addition, we propose a multimodal FedAvg algorithm to aggregate local autoencoders trained on different data modalities. We use the learned global autoencoder for a downstream classification task with the help of auxiliary labelled data on the server. We empirically evaluate our framework on different modalities including sensory data, depth camera videos, and RGB camera videos. Our experimental results demonstrate that introducing data from multiple modalities into federated learning can improve its classification performance. In addition, we can use labelled data from only one modality for supervised learning on the server and apply the learned model to testing data from other modalities to achieve decent F1 scores (e.g., with the best performance being higher than 60%), especially when combining contributions from both unimodal clients and multimodal clients

Automated Semantic Knowledge Acquisition From Sensor Data

The gathering of real-world data is facilitated by many pervasive data sources such as sensor devices and smartphones. The abundance of the sensory data raises the need to make the data easily available and understandable for the potential users and applications. Using semantic enhancements is one approach to structure and organize the data and to make it processable and interoperable by machines. In particular, ontologies are used to represent information and their relations in machine interpretable forms. In this context, a significant amount of work has been done to create real-world data description ontologies and data description models; however, little effort has been done in creating and constructing meaningful topical ontologies from a vast amount of sensory data by automated processes. Topical ontologies represent the knowledge from a certain domain providing a basic understanding of the concepts that serve as building blocks for further processing. There is a lack of solution that construct the structure and relations of ontologies based on real-world data. To address this challenge, we introduce a knowledge acquisition method that processes real-world data to automatically create and evolve topical ontologies based on rules that are automatically extracted from external sources. We use an extended k-means clustering method and apply a statistic model to extract and link relevant concepts from the raw sensor data and represent them in the form of a topical ontology. We use a rule-based system to label the concepts and make them understandable for the human user or semantic analysis and reasoning tools and software. The evaluation of our work shows that the construction of a topological ontology from raw sensor data is achievable with only small construction errors

A Linked-Data Model for Semantic Sensor Streams

This paper describes a semantic modelling scheme, a naming convention and a data distribution mechanism for sensor streams. The proposed solutions address important challenges to deal with large-scale sensor data emerging from the Internet of Things resources. While there are significant numbers of recent work on semantic sensor networks, semantic annotation and representation frameworks, there has been less focus on creating efficient and flexible schemes to describe the sensor streams and the observation and measurement data provided via these streams and to name and resolve the requests to these data. We present our semantic model to describe the sensor streams, demonstrate an annotation and data distribution framework and evaluate our solutions with a set of sample datasets. The results show that our proposed solutions can scale for large number of sensor streams with different types of data and various attributes

Discovering behavioural patterns using conversational technology for in-home health and well-being monitoring

Advancements in conversational AI have created unparalleled opportunities to promote the independence and well-being of older adults, including people living with dementia (PLWD). However, conversational agents have yet to demonstrate a direct impact in supporting target populations at home, particularly with long-term user benefits and clinical utility. We introduce an infrastructure fusing in-home activity data captured by Internet of Things (IoT) technologies with voice interactions using conversational technology (Amazon Alexa). We collect 3103 person-days of voice and environmental data across 14 households with PLWD to identify behavioural patterns. Interactions include an automated well-being questionnaire and 10 topics of interest, identified using topic modelling. Although a significant decrease in conversational technology usage was observed after the novelty phase across the cohort, steady state data acquisition for modelling was sustained. We analyse household activity sequences preceding or following Alexa interactions through pairwise similarity and clustering methods. Our analysis demonstrates the capability to identify individual behavioural patterns, changes in those patterns and the corresponding time periods. We further report that households with PLWD continued using Alexa following clinical events (e.g., hospitalisations), which offers a compelling opportunity for proactive health and well-being data gathering related to medical changes. Results demonstrate the promise of conversational AI in digital health monitoring for ageing and dementia support and offer a basis for tracking health and deterioration as indicated by household activity, which can inform healthcare professionals and relevant stakeholders for timely interventions. Future work will use the bespoke behavioural patterns extracted to create more personalised AI conversations

An adaptive method for data reduction in the Internet of Things

Enormous amounts of dynamic observation and measurement data are collected from sensors in Wireless Sensor Networks (WSNs) for the Internet of Things (IoT) applications such as environmental monitoring. However, continuous transmission of the sensed data requires high energy consumption. Data transmission between sensor nodes and cluster heads (sink nodes) consumes much higher energy than data sensing in WSNs. One way of reducing such energy consumption is to minimise the number of data transmissions. In this paper, we propose an Adaptive Method for Data Reduction (AM-DR). Our method is based on a convex combination of two decoupled Least-Mean-Square (LMS) windowed filters with differing sizes for estimating the next measured values both at the source and the sink node such that sensor nodes have to transmit only their immediate sensed values that deviate significantly (with a pre-defined threshold) from the predicted values. The conducted experiments on a real-world data show that our approach has been able to achieve up to 95% communication reduction while retaining a high accuracy (i.e. predicted values have a deviation of ±0.5 from real data values)

Ethical considerations in design and implementation of home-based smart care for dementia

It has now become a realistic prospect for smart care to be provided at home for those living with long-term conditions such as dementia. In the contemporary smart care scenario, homes are fitted with an array of sensors for remote monitoring providing data that feed into intelligent systems developed to highlight concerning patterns of behaviour or physiological measurements and to alert healthcare professionals to the need for action. This paper explores some ethical issues that may arise within such smart care systems, focusing on the extent to which ethical issues can be addressed at the system design stage. Artificial intelligence has been widely portrayed as an ethically risky technology, posing challenges for privacy and human autonomy and with the potential to introduce and exacerbate bias and inequality. While broad principles for ethical artificial intelligence have become established, the mechanisms for governing ethical artificial intelligence are still evolving. In healthcare settings the implementation of smart technologies falls within the existing frameworks for ethical review and governance. Feeding into this ethical review there are many practical steps that designers can take to build ethical considerations into the technology. After exploring the pre-emptive steps that can be taken in design and governance to provide for an ethical smart care system, the paper reviews the potential for further ethical challenges to arise within the everyday implementation of smart care systems in the context of dementia, despite the best efforts of all concerned to pre-empt them. The paper concludes with an exploration of the dilemmas that may thus face healthcare professionals involved in implementing this kind of smart care and with a call for further research to explore ethical dimensions of smart care both in terms of general principles and lived experience

Distributed spatial indexing for the Internet of Things data management

The Internet of Things (IoT) has become a new enabler for collecting real-world observation and measurement data from the physical world. The IoT allows objects with sensing and network capabilities (i.e. Things and devices) to communicate with one another and with other resources (e.g. services) on the digital world. The heterogeneity, dynamicity and ad-hoc nature of underlying data, and services published by most of IoT resources make accessing and processing the data and services a challenging task. The IoT demands distributed, scalable, and efficient indexing solutions for large-scale distributed IoT networks. We describe a novel distributed indexing approach for IoT resources and their published data. The index structure is constructed by encoding the locations of IoT resources into geohashes and then building a quadtree on the minimum bounding box of the geohash representations. This allows to aggregate resources with similar geohashes and reduce the size of the index. We have evaluated our proposed solution on a large-scale dataset and our results show that the proposed approach can efficiently index and enable discovery of the IoT resources with 65% better response time than a centralised approach and with a high success rate (around 90% in the first few attempts)

Real-time information processing of environmental sensor network data using Bayesian Gaussian processes

In this article, we consider the problem faced by a sensor network operator who must infer, in real time, the value of some environmental parameter that is being monitored at discrete points in space and time by a sensor network. We describe a powerful and generic approach built upon an efficient multi-output Gaussian process that facilitates this information acquisition and processing. Our algorithm allows effective inference even with minimal domain knowledge, and we further introduce a formulation of Bayesian Monte Carlo to permit the principled management of the hyperparameters introduced by our flexible models. We demonstrate how our methods can be applied in cases where the data is delayed, intermittently missing, censored, and/or correlated. We validate our approach using data collected from three networks of weather sensors and show that it yields better inference performance than both conventional independent Gaussian processes and the Kalman filter. Finally, we show that our formalism efficiently reuses previous computations by following an online update procedure as new data sequentially arrives, and that this results in a four-fold increase in computational speed in the largest cases considered

An Online Adaptive Algorithm for Change Detection in Streaming Sensory Data

There has been a keen interest in detecting abrupt sequential changes in streaming data obtained from sensors in wireless sensor networks for Internet of Things applications, such as fire/fault detection, activity recognition, and environmental monitoring. Such applications require (near) online detection of instantaneous changes. This paper proposes an online, adaptive filtering-based change detection (OFCD) algorithm. Our method is based on a convex combination of two decoupled least mean square windowed filters with differing sizes. Both filters are applied independently on data streams obtained from sensor nodes such that their convex combination parameter is employed as an indicator of abrupt changes in mean values. An extension of our method (OFCD) based on a cooperative scheme between multiple sensors (COFCD) is also presented. It provides an enhancement of both convergence and steady-state accuracy of the convex weight parameter. Our conducted experiments show that our approach can be applied in distributed networks in an online fashion. It also provides better performance and less complexity compared with the state-of-the-art on both of single and multiple sensors

Large-Scale Indexing, Discovery, and Ranking for the Internet of Things (IoT)

Network-enabled sensing and actuation devices are key enablers to connect real-world objects to the cyber world. The Internet of Things (IoT) consists of the network-enabled devices and communication technologies that allow connectivity and integration of physical objects (Things) into the digital world (Internet). Enormous amounts of dynamic IoT data are collected from Internet-connected devices. IoT data are usually multi-variant streams that are heterogeneous, sporadic, multi-modal, and spatio-temporal. IoT data can be disseminated with different granularities and have diverse structures, types, and qualities. Dealing with the data deluge from heterogeneous IoT resources and services imposes new challenges on indexing, discovery, and ranking mechanisms that will allow building applications that require on-line access and retrieval of ad-hoc IoT data. However, the existing IoT data indexing and discovery approaches are complex or centralised, which hinders their scalability. The primary objective of this article is to provide a holistic overview of the state-of-the-art on indexing, discovery, and ranking of IoT data. The article aims to pave the way for researchers to design, develop, implement, and evaluate techniques and approaches for on-line large-scale distributed IoT applications and services