Review on smartphone sensing technology for structural health monitoring

Sensing is a critical and inevitable sector of structural health monitoring (SHM). Recently, smartphone sensing technology has become an emerging, affordable, and effective system for SHM and other engineering fields. This is because a modern smartphone is equipped with various built-in sensors and technologies, especially a triaxial accelerometer, gyroscope, global positioning system, high-resolution cameras, and wireless data communications under the internet-of-things paradigm, which are suitable for vibration- and vision-based SHM applications. This article presents a state-of-the-art review on recent research progress of smartphone-based SHM. Although there are some short reviews on this topic, the major contribution of this article is to exclusively present a compre- hensive survey of recent practices of smartphone sensors to health monitoring of civil structures from the per- spectives of measurement techniques, third-party apps developed in Android and iOS, and various application domains. Findings of this article provide thorough understanding of the main ideas and recent SHM studies on smartphone sensing technology

Design and implementation of a convolutional neural network on an edge computing smartphone for human activity recognition

Edge computing aims to integrate computing into everyday settings, enabling the system to be context-aware and private to the user. With the increasing success and popularity of deep learning methods, there is an increased demand to leverage these techniques in mobile and wearable computing scenarios. In this paper, we present an assessment of a deep human activity recognition system’s memory and execution time requirements, when implemented on a mid-range smartphone class hardware and the memory implications for embedded hardware. This paper presents the design of a convolutional neural network (CNN) in the context of human activity recognition scenario. Here, layers of CNN automate the feature learning and the influence of various hyper-parameters such as the number of filters and filter size on the performance of CNN. The proposed CNN showed increased robustness with better capability of detecting activities with temporal dependence compared to models using statistical machine learning techniques. The model obtained an accuracy of 96.4% in a five-class static and dynamic activity recognition scenario. We calculated the proposed model memory consumption and execution time requirements needed for using it on a mid-range smartphone. Per-channel quantization of weights and per-layer quantization of activation to 8-bits of precision post-training produces classification accuracy within 2% of floating-point networks for dense, convolutional neural network architecture. Almost all the size and execution time reduction in the optimized model was achieved due to weight quantization. We achieved more than four times reduction in model size when optimized to 8-bit, which ensured a feasible model capable of fast on-device inference

Robust localization with wearable sensors

Measuring physical movements of humans and understanding human behaviour is useful in a variety of areas and disciplines. Human inertial tracking is a method that can be leveraged for monitoring complex actions that emerge from interactions between human actors and their environment. An accurate estimation of motion trajectories can support new approaches to pedestrian navigation, emergency rescue, athlete management, and medicine. However, tracking with wearable inertial sensors has several problems that need to be overcome, such as the low accuracy of consumer-grade inertial measurement units (IMUs), the error accumulation problem in long-term tracking, and the artefacts generated by movements that are less common. This thesis focusses on measuring human movements with wearable head-mounted sensors to accurately estimate the physical location of a person over time. The research consisted of (i) providing an overview of the current state of research for inertial tracking with wearable sensors, (ii) investigating the performance of new tracking algorithms that combine sensor fusion and data-driven machine learning, (iii) eliminating the effect of random head motion during tracking, (iv) creating robust long-term tracking systems with a Bayesian neural network and sequential Monte Carlo method, and (v) verifying that the system can be applied with changing modes of behaviour, defined as natural transitions from walking to running and vice versa. This research introduces a new system for inertial tracking with head-mounted sensors (which can be placed in, e.g. helmets, caps, or glasses). This technology can be used for long-term positional tracking to explore complex behaviours

Efficient Deep Clustering of Human Activities and How to Improve Evaluation

There has been much recent research on human activity re\-cog\-ni\-tion (HAR), due to the proliferation of wearable sensors in watches and phones, and the advances of deep learning methods, which avoid the need to manually extract features from raw sensor signals. A significant disadvantage of deep learning applied to HAR is the need for manually labelled training data, which is especially difficult to obtain for HAR datasets. Progress is starting to be made in the unsupervised setting, in the form of deep HAR clustering models, which can assign labels to data without having been given any labels to train on, but there are problems with evaluating deep HAR clustering models, which makes assessing the field and devising new methods difficult. In this paper, we highlight several distinct problems with how deep HAR clustering models are evaluated, describing these problems in detail and conducting careful experiments to explicate the effect that they can have on results. We then discuss solutions to these problems, and suggest standard evaluation settings for future deep HAR clustering models. Additionally, we present a new deep clustering model for HAR. When tested under our proposed settings, our model performs better than (or on par with) existing models, while also being more efficient and better able to scale to more complex datasets by avoiding the need for an autoencoder

D5.1 SHM digital twin requirements for residential, industrial buildings and bridges

This deliverable presents a report of the needs for structural control on buildings (initial imperfections, deflections at service, stability, rheology) and on bridges (vibrations, modal shapes, deflections, stresses) based on state-of-the-art image-based and sensor-based techniques. To this end, the deliverable identifies and describes strategies that encompass state-of-the-art instrumentation and control for infrastructures (SHM technologies).Objectius de Desenvolupament Sostenible::8 - Treball Decent i Creixement EconòmicObjectius de Desenvolupament Sostenible::9 - Indústria, Innovació i InfraestructuraPreprin

Connectionist systems for image processing and anomaly detection

Dissertação de mestrado integrado em Engenharia InformáticaA Inteligência Artificial (IA) e a Ciência de Dados estão cada vez mais presentes no nosso quotidiano e os benefícios que trouxeram para a sociedade nos últimos anos são notáveis. O sucesso da IA foi impulsionado pela capacidade adaptativa que as máquinas adquiriram e está estreitamente relacionada com a sua habilidade para aprender. Os sistemas conexionistas, apresentados na forma de Redes Neurais Artificiais (RNAs), que se inspiram no sistema nervoso humano, são um dos mais importantes modelos que permitem a aprendizagem. Estes são utilizados em diversas áreas, como em problemas de previsão ou classificação, apresentando resultados cada vez mais satisfatórios. Uma das áreas em que esta tecnologia se tem destacado é a Visão Computacional (Computer Vision (CV)), permitindo, por exemplo, a localização de objetos em imagens e a sua correta identificação. A Deteção de Anomalias (Anomaly Detection (AD)) é outro campo onde as RNAs vêm surgindo como uma das tecnologias para a resolução de problemas. Em cada área são utilizadas diferentes arquiteturas de acordo com o tipo de dados e o problema a resolver. Combinando o processamento de imagens e a deteção de anomalias, verifica-se uma convergência de metodologias que utilizam módulos convolucionais em arquiteturas dedicadas a AD. O objetivo principal desta dissertação é estudar as técnicas existentes nestes domínios, desenvolvendo diferentes arquiteturas e modelos, aplicando-as a casos práticos de forma a comparar os resultados obtidos em cada abordagem. O caso prático principal consiste na monitorização de pavimentos rodoviários por meio de imagens para a identificação automática de áreas degradadas. Para isso, dois protótipos de software são propostos para recolher e visualizar os dados adquiridos. O estudo de arquiteturas de RNAs para o diagnóstico da condição do asfalto por meio de imagens é o foco central no processo científico apresentado. Os métodos de Machine Learning (ML) utilizados incluem classificadores binários, Autoencoders (AEs) e Variational Autoencoders (VAEs). Para os dois últimos modelos, práticas supervisionadas e não supervisionadas são também comparadas, comprovando a sua utilidade em cenários onde não há dados rotulados disponíveis. Usando o modelo VAE num ambiente supervisionado, este apresenta uma excelente distinção entre áreas de pavimentação em boas condições e degradadas. Quando não existem dados rotulados disponíveis, a melhor opção é utilizar o modelo AE, utilizando a distribuição de semelhanças das reconstruções para calcular o threshold de separação, atingindo accuracy e precision superiores a 94%). O processo completo de desenvolvimento mostra que é possível construir uma solução alternativa para diminuir os custos de operação em relação aos sistemas comerciais existentes e melhorar a usabilidade quando comparada às soluções tradicionais. Adicionalmente, dois estudos demonstram a versatilidade dos sistemas conexionistas na resolução de problemas, nomeadamente no projeto de estruturas mecânicas, possibilitando a modelação de campos de deslocamento e pressão em placas reforçadas; e na utilização de AD para identificar locais de aglomeração de pessoas através de técnicas de crowdsensing.Artificial Intelligence (AI) and Data Science (DS) have become increasingly present in our daily lives, and the benefits it has brought to society in recent years are remarkable. The success of AI was driven by the adaptive capacity that machines gained, and it is closely related to their ability to learn. Connectionist systems, presented in the form of Artificial Neural Networks (ANNs), which are inspired by the human nervous system, are one of the principal models that allows learning. These models are used in several areas, like forecasting or classification problems, presenting increasingly satisfactory results. One area in which this technology has excelled is Com puter Vision (CV), allowing, for example, the location of objects in images and their correct identification. Anomaly Detection (AD) is another field where ANNs have been emerging as one technology for problem solving. In each area, different architectures are used according to the type of data and the problem to be solved. Combining im age processing and the finding of anomalies in this type of data, there is a convergence of methodologies using convolutional modules in architectures dedicated to AD. The main objective of this dissertation is to study the existent techniques in these domains, developing different model architectures, and applying them to practical case studies in order to compare the results obtained in each approach. The major practical use case consists of monitoring road pavements using images to automatically identify degraded areas. For that, two software prototypes are proposed to gather and visualise the acquired data. Moreover, the study of ANN architectures to diagnose the asphalt condition through images is the central focus of this work. The experimented methods for AD in images include a binary classifier network as a baseline, Autoencoders (AEs) and Variational Autoen coders (VAEs). Supervised and unsupervised practises are also compared, proving their utility also in scenarios where there is no labelled data available. Using the VAE model in a supervised setting, it presents a excellent distinction between good and bad pavement areas. When labelled data is not available, using the AE and the distribution of similarities of good pavement reconstructions to calculate the threshold is the best option with both accuracy and precision above 94%. The full development process shows it is possible to build an alternative solution to decrease the operation costs relatively to expensive commercial systems and improve usability when compared with traditional solutions. Additionally, two case studies demonstrate the versatility of connectionist systems to solve problems, namely in Mechanical Structural Design enabling the modelling of displacement and pressure fields in reinforced plates; and using AD to identify crowded places through crowd-sensing techniques

Transport Mode Detection and Classification from Smartphone Sensor Data Using Convolutional Neural Networks

Transportation is a significant component of human lives and understanding how individuals travel is an essential task in many fields. Understanding the modes of transport individuals use can lead to improvements in urban planning, traffic control, human health, and environmental sciences. The goal of transport mode detection and classification is to use smartphone devices as human behavioural sensors, to detect and classify individuals movement continuously. Smartphone devices are suitable for transport mode detection, as they are proliferated in modern societies and contain sensors that are suitable for transport mode detection. These sensors include GPS, accelerometers, gyroscopes, magnetometers, barometers, or microphones. The research in this thesis will focus on transport mode detection and classification using data from motions sensors; accelerometers, gyroscopes, magnetometers, and barometers as they do not contain the sensitive private data that is collected when using GPS or microphones. Currently, there are two approaches in state of the art in transport mode detection. In the first approach, time and frequency domain features are extracted from the signals of the motion sensors and used as input to decision tree or neural network machine learning models. In the second approach, Convolutional Neural Networks extract features by finding spatial relations in the signal data and using these for classification. This thesis investigates the use of Convolutional Neural Networks, as they have shown to outperform models trained using time and frequency domain features extracted from the data in the state of the art research. This research studies the effect of different model architectures on the accuracy of Convolutional Neural Network models when using multiple different sensors as input, as well as focusing on which combinations of sensors produce optimal results. Furthermore, the focus will be evaluating the models on real-world data in order to evaluate the feasibility of deploying applications utilizing transport mode detection. This research compares an optimized model architecture along with preprocessing techniques to state of the art Convolutional Neural Network architectures on real- world data. The best baseline algorithm achieved an overall F1 score of 0.57, while the final optimized achieved an overall F1 score of 0.72 on the testing dataset. The optimal combination of motion sensors is with the accelerometer, gyroscope, and barometer